diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml index cc6fab6c0..c80646a31 100644 --- a/.github/workflows/integration-tests.yml +++ b/.github/workflows/integration-tests.yml @@ -23,6 +23,9 @@ name: integration-tests # persistence + api-contract corpora # java-slow — the Java and Kotlin integration modules # csharp / python — that port's integration corpora +# cube — the cube-model live check: the TS generator's output in a real Cube over +# its own throwaway Postgres (FR-044). It brings its own containers on a +# private network and ignores the shared sidecar below. # # RELEASE BACKSTOP. The PRIMARY gate for the migrate-ts real-PG suites is local-ci.yml's # ts-slow lane, on every push to main; the ts-slow job here is the cold-environment @@ -46,7 +49,7 @@ jobs: strategy: fail-fast: false matrix: - lane: [ts-slow, csharp, java-slow, python] + lane: [ts-slow, csharp, java-slow, python, cube] # A single job-level Postgres sidecar shared by every lane, instead of each # port booting (and pulling) its own container per scenario. Each port's PG # helper, when it sees METAOBJECTS_TEST_PG_URL, connects to this sidecar and @@ -73,8 +76,8 @@ jobs: with: persist-credentials: false - - name: Set up Bun (TS lane) - if: matrix.lane == 'ts-slow' + - name: Set up Bun (TS lanes) + if: matrix.lane == 'ts-slow' || matrix.lane == 'cube' uses: oven-sh/setup-bun@v2 with: # Pin a known-good Bun (1.3.8 segfaults on exit; see the flake that cost a @@ -82,8 +85,8 @@ jobs: # output — the actions/cache step below does that. bun-version: '1.3.14' - - name: Cache Bun install cache (TS lane) - if: matrix.lane == 'ts-slow' + - name: Cache Bun install cache (TS lanes) + if: matrix.lane == 'ts-slow' || matrix.lane == 'cube' uses: actions/cache@v4 with: path: ~/.bun/install/cache diff --git a/AGENTS.md b/AGENTS.md index 864f9ce98..bebac3197 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,7 +22,7 @@ The first four ship per-language today across the five ports (TS / C# / Java / P ## Status -_Last refreshed 2026-10-03._ +_Last refreshed 2026-10-10._ **1.0 gating — the quiet period is RETIRED (2026-09-06).** `docs/1.0-readiness.md` §G3 no longer asks for "one coordinated release with no metamodel-breaking change." It measured a @@ -75,14 +75,14 @@ PyPI has had no product change since `0.25.0` — nothing is broken. - **Kotlin** — `codegen-kotlin` (KotlinPoet on JVM): entity + Exposed table + Spring controller + payload + relations + filter allowlist + validator + stored-proc + output-parser generators. `integration-tests-kotlin` runs the persistence-conformance corpus through Exposed against Testcontainers Postgres. **Cross-port conformance corpora** (every port runs the shared corpus): -- Metamodel: `fixtures/conformance/` (374 fixtures; 28 shared corpora in total — per-corpus counts + the corpus x port matrix live in `docs/CONFORMANCE.md`). TS / C# / Java / Python all green. +- Metamodel: `fixtures/conformance/` (374 fixtures; 29 shared corpora in total — per-corpus counts + the corpus x port matrix live in `docs/CONFORMANCE.md`). TS / C# / Java / Python all green. - Render: `fixtures/render-conformance/`. TS / C# / Java / Kotlin / Python byte-identical. - Persistence: `fixtures/persistence-conformance/`. **Query** scenarios run on every port (TS / C# / Java / Kotlin / Python), each provisioning its test DB by executing the committed, TS-produced `canonical/schema.postgres.sql` (Postgres only — Derby dropped for the cross-port query corpus, ADR-0015). The **migration** scenarios are exercised by **TS only** (TS owns schema migrations). **The corpus now gates WRITES, not just reads (SP-H):** an `op: roundtrip` scenario type INSERTs through each port's runtime/ORM write codec (NOT raw SQL), reads the row back, and asserts the wire-normalized value. The `AllTypes` entity (`roundtrip-all-types.yaml`) carries one field of **every** persistable `field.*` subtype — string/int/long/double/float/decimal/boolean/date/time/timestamp(+tz)/currency/enum/uuid/object — plus an **array-of-VO** `field.object @isArray @storage:jsonb` column (`labels`, written as 2-element / empty-`[]` / single-element arrays across the three rows) — so every subtype write+read (incl. the array-of-value-object jsonb codec) round-trips through every port against Testcontainers PG. (`field.byte`/`field.short`/`field.class` were cut as non-functional registration-only stubs — the matrix tracks only genuinely-supported subtypes; see `fixtures/registry-conformance/README.md` → "Per-subtype write-round-trip matrix".) - API-contract: `fixtures/api-contract-conformance/`. TS / C# / Java / Kotlin / Python all green — each port runs **two lanes**: a hand-rolled reference server AND its **generated** API artifact booted over HTTP (the deployed controller/routes; TS+C# full-stack vs Testcontainers PG, Java/Kotlin/Python generated controller + in-memory repo behind the consumer seam). The generated fan-out found 10 real deployment bugs golden snapshots missed. Three sub-corpora run the **generated lane only, on all five ports** — `write-through/`, `projection/` (F22: a view-only `object.projection` serves GET list + GET by id and answers every write verb with `405 {"error": "method_not_allowed"}`) and `report/` (FR-044: a view-backed `object.report` serves GET list, answers `POST` with that same `405`, and mounts no `/{id}` route). That is deliberate, not a gap: what is under test is whether a port's GENERATOR emits those routes, and a hand-rolled reference server would answer every scenario by construction. The `m2m/` sub-corpus also gates **TPH x M:N together** (base-declared, subtype-declared, abstract-mid-declared, a non-subtype source onto a subtype TARGET, and the cross-subtype source id answering `200 []`) — the two corpora were originally built disjoint (`tph/` had no relationships, `m2m/` no discriminators), which is precisely how that defect class survived. - YAML / verify corpora green across the ports that ship those layers. - **Codegen-compile gate** (all five ports; a GATE, not a corpus — it has no fixtures of its own and no row in the matrix). Every corpus above gates BEHAVIOUR; none asks whether the emitted code BUILDS, which is how four "generated code does not compile" defects shipped in 1.0.4 with the whole matrix green — `gen` exits 0 in all four cases and the adopter's build is the first thing that disagrees. Each port generates from `fixtures/persistence-conformance/canonical/meta.fitness.json` (reused deliberately: a second kitchen sink would drift from the one the other corpora already maintain) and compiles the emitted tree with its real compiler — `ts.createProgram` / Roslyn / `javac` / `KotlinCompilation` / (Python, having no static compiler) importing the generated package plus `ruff` F821. **Every port excludes its framework-bound route tier** (TS `routesFile`, C# `RoutesGenerator`, Java `SpringControllerGenerator`, Kotlin `KotlinSpringControllerGenerator`): those imports are not on an in-memory compile's classpath and stubbing them drowns the signal, so that tier is proven by the api-contract integration lane instead. One cross-port rule, not four local concessions. Found 5 further real defects on first run. Boundary detail: `docs/CONFORMANCE.md` → "Split coverage". -**Key cross-language features shipped:** FR5 family (a/b/c/d/e + WARN envelope-shape — actionable loader errors per ADR-0009); FR-003 (Java RDB runtime persistence + projections; schema migrations are TS-only — the Java migration engine was removed); FR-006 (template.output parser-on-receipt codegen per ADR-0010 in all 5 ports); FR-008 + FR-009 (cross-port REST API contract + the nine filter operators); FR-018 (M:N relationship codegen in all 5 ports — entity navigation + idiomatic ORM wiring [Drizzle m2m / EF Core `UsingEntity` / Spring repo+JPA / Exposed / Pydantic+route as the SQLAlchemy-secondary equivalent] + REST traversal `GET //{id}/` + Tier-2 docs, gated by the shared api-contract m2m corpus in both lanes + persistence-conformance; the TanStack M:N client hook is a deferred client-ergonomics follow-up); SP-H (field-subtype end-to-end hardening: every concrete `field.*` subtype write+read round-trips cross-port via the persistence `op: roundtrip` gate; cut `field.byte`/`field.short`/`field.class` non-functional stubs; cross-port filter-op reconciliation for uuid/currency); source v2 paradigm (ADR-0007); metadata-ktx Kotlin facade; per-target output directories (TS codegen); FR-044 (the reporting vocabulary — `dimension.attribute`/`dimension.time`, `measure.aggregate`/`measure.ratio`, `segment.filter`, `object.report`, `@spine` on a report to take rows from a to-one dimension entity, `@default` on a measure for empty aggregates, plus the relative-date filter value — registered and loader-validated in all five ports; a report that declares a read-only `source.rdb @kind: view` is lowered to a SQL view by `meta migrate` and read by every port, and served by a generated read-only list route in every port, gated by the api-contract `report/` sub-corpus; no client hook, grid or form is generated for a report yet, see [docs/features/reporting.md](docs/features/reporting.md)). +**Key cross-language features shipped:** FR5 family (a/b/c/d/e + WARN envelope-shape — actionable loader errors per ADR-0009); FR-003 (Java RDB runtime persistence + projections; schema migrations are TS-only — the Java migration engine was removed); FR-006 (template.output parser-on-receipt codegen per ADR-0010 in all 5 ports); FR-008 + FR-009 (cross-port REST API contract + the nine filter operators); FR-018 (M:N relationship codegen in all 5 ports — entity navigation + idiomatic ORM wiring [Drizzle m2m / EF Core `UsingEntity` / Spring repo+JPA / Exposed / Pydantic+route as the SQLAlchemy-secondary equivalent] + REST traversal `GET //{id}/` + Tier-2 docs, gated by the shared api-contract m2m corpus in both lanes + persistence-conformance; the TanStack M:N client hook is a deferred client-ergonomics follow-up); SP-H (field-subtype end-to-end hardening: every concrete `field.*` subtype write+read round-trips cross-port via the persistence `op: roundtrip` gate; cut `field.byte`/`field.short`/`field.class` non-functional stubs; cross-port filter-op reconciliation for uuid/currency); source v2 paradigm (ADR-0007); metadata-ktx Kotlin facade; per-target output directories (TS codegen); FR-044 (the reporting vocabulary — `dimension.attribute`/`dimension.time`, `measure.aggregate`/`measure.ratio`, `segment.filter`, `object.report`, `@spine` on a report to take rows from a to-one dimension entity, `@default` on a measure for empty aggregates, plus the relative-date filter value — registered and loader-validated in all five ports; a report that declares a read-only `source.rdb @kind: view` is lowered to a SQL view by `meta migrate` and read by every port, and served by a generated read-only list route in every port, gated by the api-contract `report/` sub-corpus; the TypeScript `cube-model` reference generator writes the vocabulary as Cube data-model files, gated by the `fixtures/cube-model/` corpus and a `cube` lane that loads the output into a real Cube; no grid or form is generated for a report yet, see [docs/features/reporting.md](docs/features/reporting.md) and [docs/features/cube-export.md](docs/features/cube-export.md)). **Latest release: 1.0.13** (2026-10-03) — npm `1.0.13`, PyPI `1.0.13`, NuGet `1.0.13`, Maven Central `8.0.13`. A PATCH: an already-plural entity name (`Stats`, `Settings`) no longer doubles in API-surface names (REST paths, hooks, finders, DbSets) in any port, while default physical table names stay frozen on the old rule; two entities that would share one API name are now a generation error; two Kotlin controller compile fixes (`field.inet` filter ops, `@dbColumnType: uuid` on a string field). Gated by a private `1.0.13-rc.1` build on the adopter estate (`rc-gate.sh` 7/7) and a full `--strict-toolchains` local CI run. The previous release, 1.0.12 (2026-10-02), added `fmt` in every CLI, a deprecated-reference `verify` advisory, the `onLocate` extract hook, and an Exposed 1.x Kotlin output mode. diff --git a/CHANGELOG.md b/CHANGELOG.md index 067b42218..b6911c971 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,41 @@ it until 1.1 ships._ ### Added +- **TypeScript: the `cube-model` reference generator (FR-044 Plan 4).** `cubeModel()` writes the + reporting vocabulary as Cube data-model files, `model/cubes/.yml`: a cube for each + table-backed entity that declares dimensions, measures or segments, a join for each to-one + reference between cubes, dimensions (a time dimension carries its `@grains` as + `meta.grains`), measures, segments, and a `rollup` pre-aggregation for each served report. A + report with a relative date (in its `@filter`, its `@segment` or a listed measure's condition) + gets no rollup, and a `Scope` segment when it has a `@filter`. A served report that + declares `@spine` is a Cube view, `model/views/.yml`, rooted at the spine cube over a + `public: false` `Facts` cube whose own `sql` holds the report's scope (so a spine row + whose facts are all filtered out keeps its row), with no rollup; the ordinary cubes gain no + reverse join. It is opt-in + (`meta init` wires nothing), listed by `meta gen --list` in the `capability` layer, ejectable + (`meta eject cube-model`) and drift-checked by `meta verify --codegen`. It writes Postgres + and MySQL SQL and refuses `sqlite` and `d1`. What Cube cannot hold is an `ERR_CUBE_*` + generation error that names the node; the exporter never renames or drops. **No vocabulary + change:** `metamodelVersion` stays 1.1 and `expected-registry.json` is untouched. The report + view lowering and the exporter now share one SQL module, moved without changing a byte of + any view. A measure's `@default` is a `public: false` `Raw` aggregate and the measure as + `COALESCE({Raw}, n)` (a ratio's default wraps its quotient, and an operand's own default + reaches the ratio), so Cube reads the default where the view does. A MySQL tuple distinct + count is the view's own `COUNT(DISTINCT a, b)`. Gated by the new `fixtures/cube-model/` corpus + (44 cases and a canonical golden; the 29th shared corpus in `docs/CONFORMANCE.md`, run by + TypeScript only). + See [docs/features/cube-export.md](docs/features/cube-export.md). +- **A `cube` lane checks the exporter against a real Cube.** `scripts/ci-local.sh --only cube` + (also `scripts/integration-test.sh cube`, `bun run test:cube` in `integration-tests`, and a + `cube` entry in `integration-tests.yml`) loads the canonical model's output into + `cubejs/cube:v1.7.43` over a private `postgres:16-alpine`, requires the Cube query for each + served report to return the rows of its view (a `@spine` report through its Cube view, its + empty spine rows included), requires each of the 34 corpus cases that hold a tree to compile + (the two MySQL ones included), and reads the escaping case's literals and a ratio over a + defaulted operand back from Cube's `/v1/sql`. It owns a private Docker network and an ephemeral `127.0.0.1` port and + never uses the shared Postgres sidecar. Without Docker it is a SKIP behind a banner (a failure + under `--strict-toolchains`). The full `scripts/ci-local.sh` runs it; `--quick` and + `--no-integration` do not. - **Python: a run-time validator runner, `run_validators`.** `metaobjects.runtime.run_validators(entity, data)` validates a data mapping against an entity's metadata with no generated code and no database, and `ObjectManager.validate(entity_name, data)` does the same for a loaded entity. It is the diff --git a/README.md b/README.md index 36beca94d..2ce6d7795 100644 --- a/README.md +++ b/README.md @@ -154,7 +154,7 @@ first-week wedge plan — and `meta init` picks up from there. | Template-drift verify | Yes | Yes (`Verify.check`) | Yes (via Java) | Yes (`dotnet meta verify`) | Yes (`metaobjects.render.verify`) | | YAML authoring (sigil-free → JSON) | Yes | Yes | Yes (via Java) | Yes | Yes | | Capability requirements (`requirement.*`) | `meta verify` gate + authoring lint; `requirementTests()` stubs | `mvn metaobjects:verify` gate; JUnit Jupiter `requirement-tests` | Gate via the Java Maven goal; JUnit Jupiter `requirement-tests` | `dotnet meta verify` gate; xUnit `requirement-tests` | `metaobjects verify` gate; pytest `requirement-tests` | -| Reporting vocabulary (`dimension` / `measure` / `segment` / `object.report`, FR-044) | Registered; a view-backed report becomes a SQL view in `meta migrate`, is read by `ObjectManager`, and gets generated read-only list routes (no client hook yet) | Registered; OMDB reads a view-backed report; generates a read-only Spring controller, DTO, repository and filter allowlist | Registered (via Java); generates an Exposed table object, a row data class, a filter allowlist and a read-only Spring controller per view-backed report | Registered; generates a keyless EF Core row type, read-only routes and a filter allowlist per view-backed report | Registered; `ObjectManager` reads a view-backed report; generates a Pydantic row model, a read-only FastAPI router and a filter allowlist | +| Reporting vocabulary (`dimension` / `measure` / `segment` / `object.report`, FR-044) | Registered; a view-backed report becomes a SQL view in `meta migrate`, is read by `ObjectManager`, and gets generated read-only list routes and a TanStack list hook (no grid or form); the `cube-model` generator writes the vocabulary as Cube data-model files | Registered; OMDB reads a view-backed report; generates a read-only Spring controller, DTO, repository and filter allowlist | Registered (via Java); generates an Exposed table object, a row data class, a filter allowlist and a read-only Spring controller per view-backed report | Registered; generates a keyless EF Core row type, read-only routes and a filter allowlist per view-backed report | Registered; `ObjectManager` reads a view-backed report; generates a Pydantic row model, a read-only FastAPI router and a filter allowlist | | Libraries (`libraries: [...]`) | Yes | Yes | Yes (via Java) | Yes | Yes | | Metadata dependencies (`dependencies`) | Yes (`meta deps sync`, `path` transport) | Phase 2 | Phase 2 | Phase 2 | Yes (loads the synced snapshot) | | Runtime metadata (ObjectManager-style) | Yes (`runtime-ts`) | Yes (OMDB) | Yes (via Java OMDB + Exposed) | Roadmap | Yes (ObjectManager) | diff --git a/agent-context/skills/metaobjects-authoring/references/reporting.md b/agent-context/skills/metaobjects-authoring/references/reporting.md index 47c54d6d8..6db3b1472 100644 --- a/agent-context/skills/metaobjects-authoring/references/reporting.md +++ b/agent-context/skills/metaobjects-authoring/references/reporting.md @@ -155,6 +155,20 @@ A concrete report whose read source is `@kind: view` is served by every port's g The files each port generates are in the `metaobjects-codegen` skill's reference for that language ("Reports"). +## Exporting to Cube + +The TypeScript `cube-model` generator (`meta eject cube-model`; see the `metaobjects-codegen` skill's +TypeScript reference) writes this vocabulary as Cube data-model files: a cube for each table-backed +entity that declares dimensions, measures or segments, and a rollup for each served report. It keeps +your names as written, so an entity, dimension, measure or segment name must be one Cube accepts (a +letter first, then letters, digits and `_`, and not a Python keyword such as `from`, `class` or `in`); +otherwise the export fails with `ERR_CUBE_INVALID_NAME` and renames nothing. It also refuses a +dimension over an array, object or map field. A report with a relative date in its `@filter`, +its `@segment`'s filter or a listed measure's condition gets no rollup. Its `Scope` +segment is written only when it has a `@filter`. A served report with `@spine` becomes a Cube view +with its scope inside a facts cube and no rollup, so the spine's empty rows survive, and a measure's +`@default` is a `COALESCE`, as in the view. A report that is not served contributes nothing. Reference: `docs/features/cube-export.md`. + ## What a report does not have In TypeScript a served report gets a generated list hook; no port generates a grid or a form for a report, and the other ports have no client tier, so there you get the route and the row type. There is no `measure.derived`, no query-time choice of dimensions or measures, and no time-zone vocabulary. diff --git a/agent-context/skills/metaobjects-codegen/references/typescript.md b/agent-context/skills/metaobjects-codegen/references/typescript.md index 13cfc12ac..0462bfdd7 100644 --- a/agent-context/skills/metaobjects-codegen/references/typescript.md +++ b/agent-context/skills/metaobjects-codegen/references/typescript.md @@ -190,6 +190,25 @@ report hook until you resync it (`meta eject`). On SQLite a decimal field of a r mount. A report with no view source, or an abstract one, generates nothing. The route and contract: `references/reporting.md` in the `metaobjects-authoring` skill. +**Cube export (`cubeModel()`).** A reference helper that writes the reporting vocabulary +(`dimension.*`, `measure.*`, `segment.filter`, a served `object.report`) as Cube data-model files, +`model/cubes/.yml`: a cube for each table-backed entity that declares any of it, a join for +each to-one reference between cubes, and a `rollup` pre-aggregation on the `@from` cube for each +served report (a report with a relative date in its `@filter`, `@segment` or a listed measure's +condition gets no rollup, and a `Scope` segment only when it has a `@filter`). It +is opt-in: `meta eject cube-model` copies it, or import `cubeModel` from +`@metaobjectsdev/codegen-ts` (also from `/generators`), and point it at the Cube project with a +target (`targets: { cube: { outDir: "cube" } }`, `cubeModel({ target: "cube" })`). Options: +`dialect` (`"postgres"` or `"mysql"`, default the config's; `sqlite` and `d1` raise +`ERR_CUBE_UNSUPPORTED_DIALECT`), `filter` and `target`. What Cube cannot take is an `ERR_CUBE_*` +error naming the node (an array or object dimension, a name that is a Python keyword, two members +of one name); the exporter never renames or drops. A measure's `@default` is a `COALESCE` over it +(a `measure.aggregate` keeps its aggregate as a `public: false` `Raw` member), and a served +report with `@spine` is a Cube view, `model/views/.yml`, rooted at the spine cube over a +`public: false` `Facts` cube that holds the report's scope, with no rollup. The mapping, +wiring and the `cube` lane that checks it against a real Cube are in +`docs/features/cube-export.md`. + The `CREATE VIEW` DDL is generated by `meta migrate` from the projection's `origin.*` children — `origin.passthrough` (a forwarded column), `origin.aggregate` (`@agg` `count`/`sum`/`avg`/`min`/`max`, plus the #195 `any`/`all` diff --git a/docs/CONFORMANCE.md b/docs/CONFORMANCE.md index 41dfec3fa..3a2c51eff 100644 --- a/docs/CONFORMANCE.md +++ b/docs/CONFORMANCE.md @@ -1,6 +1,6 @@ # Conformance coverage -The MetaObjects standard ships **28 shared conformance corpora** under +The MetaObjects standard ships **29 shared conformance corpora** under [`fixtures/`](../fixtures/). Every port runs every corpus that is *applicable to it* and asserts the same expected behaviour against the same fixtures. **This page is the inverse index**: fixture → feature doc + per-port pass status, and it is the @@ -54,6 +54,7 @@ regenerate with `ls -d fixtures//*/ | wc -l` for directory-shaped corpor | [`fixtures/requirement-test-identity-conformance/`](../fixtures/requirement-test-identity-conformance/) (the identity, skip state and digest of each generated requirement test, and the filter seam, ADR-0057) | 26 cases | ✓ (reference) | ✓ | identity function inherits via Java; emitted names asserted by its own generator test | ✓ | ✓ | | [`fixtures/naming-conformance/`](../fixtures/naming-conformance/) | 8 cases | ✓ | ✓ | inherits via Java (`RouteNaming.pluralize`) | ✓ | ✓ | | [`fixtures/codegen-noop/`](../fixtures/codegen-noop/) (FR-044 — reporting vocabulary: what is lowered, what stays inert) | 1 model pair (`reporting/with` vs `reporting/without`) | ✓ (codegen + migrate) | ✓ | ✓ | ✓ | ✓ | +| [`fixtures/cube-model/`](../fixtures/cube-model/) (FR-044 Plan 4 — the `cube-model` generator's golden Cube files, plus the canonical model's golden) | 44 cases (34 trees + 10 errors) + 1 canonical golden | ✓ (the exporter is TypeScript-only, spec R6; its `cube` lane also loads the files into a real Cube) | — | — | — | — | A ✓ means the port runs the corpus green; an explicit `n / m` is used where a port carries a ledgered divergence. The two ledgered YAML fixtures are documented @@ -82,7 +83,8 @@ the corpora above do two different jobs. Only the first is a promise to adopters A red cell here is a MetaObjects bug. - **Template quality checks — not a promise.** The generated lane of `api-contract-conformance/`, `generator-registry-conformance/` (stable generator names), - `codegen-noop/` (vocabulary with no lowering yet emits nothing), + `codegen-noop/` (vocabulary with no lowering yet emits nothing), `cube-model/` (the Cube + exporter's output, a TypeScript reference helper), `requirement-test-identity-conformance/` (which tests the `requirement-tests` reference generator yields from a ledger and what each is called; the text it writes is not pinned) and the codegen-compile gate. They check that the reference generators are correct @@ -123,6 +125,27 @@ Stated as mechanisms rather than as a list of attribute names on purpose — the requirement vocabulary has a breaking change scheduled (FR-038), which moves what the manifest contains without moving these boundaries. +**The Cube exporter** ([features/cube-export.md](features/cube-export.md)): + +- *Mapping — gated in TypeScript only, by decision.* `fixtures/cube-model/` holds 44 cases, 34 + with the exact tree the generator writes and 10 with the exact error message, plus the + canonical model's golden. `codegen-ts/test/cube/cube-model-corpus.test.ts` and + `cube-model-canonical.test.ts` run them. No other port has an exporter (spec R6), so no other + port makes a claim the corpus would have to check. +- *Cube accepts the output — queried on Postgres, compiled in both dialects.* The `cube` lane + (`scripts/ci-local.sh --only cube`, which needs Docker) loads the canonical golden and the 34 + cases that hold a tree into a pinned Cube (`cubejs/cube:v1.7.43`) and requires each to + compile, the two MySQL cases included: compiling a model runs no SQL, so the lane's Postgres + data source serves them too. For the nine canonical reports it compares the Cube query with the + report view (a `@spine` report through its Cube view, its empty spine rows included), and for + the `escaping` and `measure-default` cases it reads the SQL back from Cube's `/v1/sql`. No + SQL of a MySQL case is executed; the goldens hold it. The lane runs Cube in development mode, + so Cube's production mode with a separate Cube Store is not gated. +- *The lane is a gate, not a second corpus.* Like the codegen-compile gate, it reuses + `fixtures/persistence-conformance/canonical/meta.fitness.json` and the committed + `schema.postgres.sql` rather than adding a model beside them; only the seed rows are its own + (`fixtures/cube-model/canonical/seed.sql`). + **Does the generated code compile** (`codegen-compile-conformance`, all five ports): - *Not a corpus, so it has no row.* It reuses @@ -458,12 +481,27 @@ outside ASCII. The case list is in the Kotlin's identity function is the Java one; the names its generator emits are asserted by `server/java/codegen-kotlin/src/test/kotlin/com/metaobjects/generator/kotlin/KotlinRequirementTestsGeneratorTest.kt`. +### `fixtures/cube-model/` (44 cases) + +All 44 cases and the canonical golden → [features/cube-export.md](features/cube-export.md). +Each case is the smallest model for one mapping rule (an entity's cube, a join, an alias cube, a +dimension type, a measure, a measure's `@default`, a segment, a relative date, a rollup, a +`@spine` report's Cube view, the escaping rules) or for one +`ERR_CUBE_*` refusal, and holds exactly one of an expected file tree and an expected error +message. The case list, with the rule each pins, is the +[corpus README](../fixtures/cube-model/README.md). + +**One port runs it:** TypeScript, in +`server/typescript/packages/codegen-ts/test/cube/cube-model-corpus.test.ts`. The `cube` lane +(`server/typescript/packages/integration-tests/cube-live/cube-model.live.ts`) loads the same +files into a real Cube. + ## Orphaned fixtures (tested but not yet documented) The fixtures in the nine corpora mapped above (metamodel 374 + yaml 16 + verify 31 + render 15 + persistence 42 + api-contract 82 + source-resolution 25 + scope 10 + -dependency 23) each map to a feature doc, and so do the two requirement corpora, whose -case lists live in their own READMEs. None are orphaned today. The remaining +dependency 23) each map to a feature doc, and so do the two requirement corpora and the +`cube-model` corpus, whose case lists live in their own READMEs. None are orphaned today. The remaining corpora in the totals table gate tooling contracts (registry manifests, provider composition, agent context, docs emit) rather than user-facing metamodel behaviour, so they have no feature-doc row. diff --git a/docs/README.md b/docs/README.md index ea3574c14..d0ee9dcab 100644 --- a/docs/README.md +++ b/docs/README.md @@ -41,6 +41,7 @@ docs/ │ ├── metadata-dependencies.md # building on a metadata model published elsewhere (FR-023) │ ├── libraries.md # reusable declared design you opt into, a layer at a time (FR-043) │ ├── reporting.md # dimensions, measures, segments, reports: vocabulary and load rules (FR-044) +│ ├── cube-export.md # the cube-model generator: reporting vocabulary as Cube data-model files (FR-044) │ └── own-your-codegen.md # scaffold-and-own generator ownership (ADR-0034) └── ports/ # one file per language/framework port ├── typescript.md @@ -67,6 +68,7 @@ this tree is documentation, not the source of truth. | Adopt a design MetaObjects already ships — users/groups/roles, an LLM trace envelope — instead of authoring it (`libraries`, `meta eject `) | [`features/libraries.md`](features/libraries.md) | | Record what the system is supposed to do, and stop agents reviving retired features | [`features/requirements.md`](features/requirements.md) | | Declare what a dashboard groups by and counts (`dimension`, `measure`, `segment`, `object.report`; load-time checked; a view-backed report becomes a SQL view served by a generated read-only list route and a TypeScript list hook) | [`features/reporting.md`](features/reporting.md) | +| Export the reporting vocabulary to Cube (`cube-model`: cubes, joins, rollups; the `cube` lane that checks it against a real Cube) | [`features/cube-export.md`](features/cube-export.md) | | Wire prompt construction (FR-004) | [`features/templates-and-payloads.md`](features/templates-and-payloads.md) | | Share a metadata shape across multiple instances (abstracts, `extends:`) | [`features/abstracts-and-inheritance.md`](features/abstracts-and-inheritance.md) | | Add a custom metamodel subtype or attribute to a downstream project | [`features/extending-with-providers.md`](features/extending-with-providers.md) + [`recipes/extending-metaobjects-with-providers.md`](recipes/extending-metaobjects-with-providers.md) | diff --git a/docs/features/cube-export.md b/docs/features/cube-export.md new file mode 100644 index 000000000..1eadb2564 --- /dev/null +++ b/docs/features/cube-export.md @@ -0,0 +1,652 @@ +# Cube export: the `cube-model` generator + +_`cubeModel()` — write the reporting vocabulary of a model as Cube data-model files: cubes, +joins, dimensions, measures, segments, and a rollup for each served report._ + +**Status:** a TypeScript reference helper (FR-044 Plan 4; +[ADR-0034](../../spec/decisions/ADR-0034-codegen-scaffold-and-own.md) Amendment 3). It is +listed by `meta gen --list`, copied into your repo with `meta eject cube-model`, and checked +by `meta verify --codegen`. It is not core: what MetaObjects guarantees is the vocabulary and +the loader's checks on it, and a generator that writes files into your repo is a helper you can +copy and own. It adds no vocabulary (`metamodelVersion` stays 1.1) and has no runtime: the +files it writes import nothing. + +**What it does.** For each concrete entity that has a table and declares a `dimension`, +`measure` or `segment`, it writes `model/cubes/.yml`. The file holds one cube: the +table, the joins to the cubes its references reach, and its dimensions, measures and +segments. For each served report it adds a `rollup` pre-aggregation to the report's `@from` +cube, and for a served report that declares `@spine` it writes a Cube view, +`model/views/.yml`, instead. Cube reads these files from its own project. + +**What it does not do.** It never talks to Cube or to a database. There is no dbt MetricFlow +exporter (it waits for the first adopter who asks, spec decision D5). No other port has an +exporter (spec R6): the files are language-neutral YAML, and the Node `meta` CLI writes them. + +**Entirely opt-in.** `meta init` wires no generator. A project that does not configure +`cube-model` gets no file. A model that declares no dimension, measure or segment gets no file +even when the generator is configured. + +## What it writes + +Paths are relative to the generator's target `outDir`. Every file starts with +`# @generated by @metaobjectsdev/codegen-ts — cube-model`. + +| The model declares | The run writes | +|---|---| +| a concrete entity with a writable `source.rdb` table that declares, or inherits through `extends`, at least one `dimension`, `measure` or `segment` | `model/cubes/.yml`, the entity's cube. Its rollups and report scope segments are in the same file. | +| an entity that a dimension's `@via` reaches and that is not a cube already | a **join-target cube**, `model/cubes/.yml`, with `public: false`: its primary key and the members the reaching dimensions read | +| two or more to-one hops from one cube onto the same entity, or a hop onto the cube's own entity | an **alias cube** for each hop, `model/cubes/_.yml`, with `public: false` | +| a TPH subtype that declares reporting vocabulary | a cube whose `sql` is a `SELECT` over the base table with the subtype's discriminator predicate, in place of `sql_table` | +| an abstract entity | nothing. Its members land on each concrete entity that inherits them. | +| an entity with reporting vocabulary and no table | nothing (it is inert, like any object with no source) | +| a served report | no file. In the cube of its `@from` entity: a rollup, unless the report holds a relative date anywhere (then it has none, see [Reports](#reports-rollups-and-scope-segments)), and a scope segment when it has a `@filter`. | +| a served report that declares `@spine` | a **Cube view**, `model/views/.yml`; its **facts cube**, `model/cubes/Facts.yml`, `public: false`; for a spine of more than one hop, a **chain cube** `model/cubes/_.yml`, `public: false`, for each entity between; and one `one_to_many` join on the spine cube. No rollup and no scope segment (see [Reports with `@spine`](#reports-with-spine-a-cube-view)). | +| a report with no `source.*`, an abstract report, or one whose read source is not `@kind: view` | nothing | +| none of the reporting vocabulary | no file | + +A report is served when it is concrete and its read source has `@kind: view`, the rule every +port's route generators use ([How a report is served](reporting.md#how-a-report-is-served)). + +## Wiring it + +Cube reads its model from `model/cubes/` in its own project, so give the generator a target +that points there: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; +// Copied in by `meta eject cube-model`: yours to edit (ADR-0034). +import { cubeModel } from "./codegen/generators/cube-model.js"; + +export default defineConfig({ + outDir: "src/generated", + dialect: "postgres", + targets: { cube: { outDir: "cube" } }, // the Cube project: files land in cube/model/cubes/ + generators: [/* ... */ cubeModel({ target: "cube" })], +}); +``` + +`meta eject cube-model` copies one file, `codegen/generators/cube-model.ts`, and prints the +import and the entry to wire. It copies nothing into `codegen/runtime/`, because the output +imports nothing. What you own in the copy is which entities get a cube, where the files land +and the YAML call. The mapping (`buildCubeModel`, which raises every refusal below) and the +YAML writer (`renderCubeYaml`) stay in the package, both exported from +`@metaobjectsdev/codegen-ts`. To run the package's own copy, import `cubeModel` from +`@metaobjectsdev/codegen-ts` instead. + +| Option | Meaning | +|---|---| +| `dialect` | `"postgres"` or `"mysql"`, the database Cube reads. Default: the config's `dialect`. | +| `filter` | A predicate over entities, ANDed with the gate in the table above. It is the universe of the build: an entity it excludes gets no cube of its own, and its dimensions add no member to another cube. If another cube's `@via` reaches an excluded entity, it is still written as a join-target cube, without its own measures, segments or rollups, unless every hop onto it goes through an alias cube. | +| `target` | The named output target the files are written under. | + +The config's `columnNamingStrategy` (default `snake_case`) names the columns in the SQL, as it +does for the report views. + +**Dialects.** The exporter writes SQL for Postgres and MySQL only, the two dialects it quotes +and casts for. A config whose `dialect` is `sqlite` or `d1` raises +`ERR_CUBE_UNSUPPORTED_DIALECT`, and the message says to pass the dialect of the database Cube +reads, as in `cubeModel({ dialect: "postgres" })`. The error is raised only when the run would +write a cube file. A project with no reporting vocabulary, or a run that selects no cube, writes +nothing and raises nothing under any dialect. + +## A worked example + +This is the `Purchase` and `Program` model from [reporting](reporting.md), with two served +reports. `RevenueByProgram` groups by program title and month. `RecentRevenue` has a relative +date in its `@filter`. + +```jsonc +{ "metadata.root": { + "package": "acme::shop", + "children": [ + { "object.entity": { + "name": "Program", + "children": [ + { "source.rdb": { "@table": "programs" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "title" } }, + { "identity.primary": { "name": "id", "@fields": ["id"] } } + ] + }}, + { "object.entity": { + "name": "Purchase", + "children": [ + { "source.rdb": { "@table": "purchases" } }, + { "field.long": { "name": "id" } }, + { "field.long": { "name": "programId" } }, + { "field.string": { "name": "customerEmail" } }, + { "field.currency": { "name": "amountCents" } }, + { "field.string": { "name": "status" } }, + { "field.timestamp": { "name": "purchasedAt" } }, + { "identity.primary": { "name": "id", "@fields": ["id"] } }, + { "identity.reference": { "name": "fkProgram", "@fields": ["programId"], + "@references": "Program" } }, + { "relationship.association": { "name": "program", "@objectRef": "Program", + "@cardinality": "one" } }, + { "segment.filter": { "name": "active", "@filter": { "status": "active" } } }, + { "dimension.attribute": { "name": "programTitle", + "@of": "Program.title", "@via": "Purchase.program" } }, + { "dimension.time": { "name": "purchasedAt", "@of": "Purchase.purchasedAt", + "@grains": ["day", "month"] } }, + { "measure.aggregate": { "name": "purchases", "@agg": "count", "@of": "Purchase.id", + "@segment": "active" } }, + { "measure.aggregate": { "name": "revenue", "@agg": "sum", "@of": "Purchase.amountCents", + "@segment": "active" } } + ] + }}, + { "object.report": { + "name": "RevenueByProgram", + "@from": "Purchase", + "@dimensions": ["programTitle", "purchasedAt:month"], + "@measures": ["purchases", "revenue"], + "children": [ + { "source.rdb": { "@kind": "view", "@view": "v_revenue_by_program" } } + ] + }}, + { "object.report": { + "name": "RecentRevenue", + "@from": "Purchase", + "@measures": ["purchases", "revenue"], + "@filter": { "purchasedAt": { "gte": { "now": "-P30D" } } }, + "children": [ + { "source.rdb": { "@kind": "view", "@view": "v_recent_revenue" } } + ] + }} + ] +}} +``` + +With `dialect: "postgres"` and the default naming strategy, the generator writes two files. +`model/cubes/Purchase.yml`: + +```yaml +# @generated by @metaobjectsdev/codegen-ts — cube-model +cubes: + - name: Purchase + sql_table: '"purchases"' + joins: + - name: Program + relationship: many_to_one + sql: '{CUBE}."program_id" = {Program}."id"' + dimensions: + - name: id + sql: '{CUBE}."id"' + type: number + primary_key: true + - name: programTitle + sql: '{Program.title}' + type: string + - name: purchasedAt + sql: '{CUBE}."purchased_at"' + type: time + meta: + grains: [day, month] + measures: + - name: purchases + sql: '{CUBE}."id"' + type: count + filters: + - sql: '{CUBE}."status" = ''active''' + - name: revenue + sql: '{CUBE}."amount_cents"' + type: sum + filters: + - sql: '{CUBE}."status" = ''active''' + segments: + - name: active + sql: '{CUBE}."status" = ''active''' + - name: recentRevenueScope + sql: '{CUBE}."purchased_at" >= (now() - INTERVAL ''P30D'')' + pre_aggregations: + - name: RevenueByProgram + type: rollup + measures: [CUBE.purchases, CUBE.revenue] + dimensions: [CUBE.programTitle] + time_dimension: CUBE.purchasedAt + granularity: month +``` + +and `model/cubes/Program.yml`, which exists only because `programTitle` reads `Program.title`: + +```yaml +# @generated by @metaobjectsdev/codegen-ts — cube-model +cubes: + - name: Program + sql_table: '"programs"' + public: false + dimensions: + - name: id + sql: '{CUBE}."id"' + type: number + primary_key: true + - name: title + sql: '{CUBE}."title"' + type: string + public: false +``` + +`RevenueByProgram` became a rollup. `RecentRevenue` became the segment `recentRevenueScope` and +no rollup, because its filter holds a relative date (see [Reports](#reports-rollups-and-scope-segments)). + +## How the model maps + +Each rule below is pinned by a golden case in [`fixtures/cube-model/`](../../fixtures/cube-model/), +named in that directory's README. + +### Cubes and primary keys + +A cube is named after its entity, with no package. Its `sql_table` is the quoted table, with +the schema in front when `source.rdb` declares `@schema` (`"sales"."programs"`). It carries the +entity's `title` and `description` when it has them; `notes` is never written. + +Each field of the entity's `identity.primary` becomes a dimension named after the field, with +`primary_key: true` (composite key: one dimension per field). Cube makes a primary-key dimension +non-public by default, so no `public` key is written. + +A declared dimension without `@via` that is named after a key field and reads that field is that +key dimension, not a second one: the cube gets one dimension, with `primary_key: true`, +`public: true` (the declaration says the key is meant to be queried), and the declared +dimension's `title`, `description` and `meta.grains`. A report lists it like any dimension. A +dimension named after a key field that reads another field, or that has a `@via`, is +`ERR_CUBE_MEMBER_COLLISION`. + +A TPH subtype's cube has no `sql_table`. Its `sql` selects from the base table and applies the +subtype's discriminator, such as `SELECT * FROM "auths" WHERE "type" = 'Bridge'`. + +### Dimensions + +The `@of` field's subtype decides the Cube `type` and the `sql`, as it decides the column type +of a report view. + +| `@of` field | Cube `type` | `sql` | +|---|---|---| +| `string`, `enum` (string-backed), `uuid`, `time` (the `field.time` subtype), `uri`, `inet` | `string` | the column | +| `enum` with `@intValueMap` | `string` | `CASE WHEN 1 THEN 'LOW' … END`, so the value is the member symbol, as every port's read returns it | +| `int`, `long`, `double`, `float`, `decimal`, `currency` | `number` | the column | +| `boolean` | `boolean` | the column | +| `timestamp`, with or without `@localTime` | `time` | the column | +| `date` | `time` | `CAST( AS TIMESTAMP)`; on MySQL `CAST( AS DATETIME)`, because MySQL's `CAST` has no `TIMESTAMP` target | +| a field with `isArray`, a `field.object`, a `field.map`, or a field with `@objectRef` | none | `ERR_CUBE_UNMAPPABLE_DIMENSION`: Cube has no array or JSON dimension type | + +`` is `{CUBE}.""`, the owning cube's column, quoted unconditionally. + +A `dimension.time` carries its `@grains` as `meta: { grains: [...] }`, in declared order. Cube +offers every granularity on a time dimension and cannot restrict them, so the grains are +carried and not enforced. + +A dimension with `@via` reads a member of the cube its last hop joins, such as +`sql: '{Program.title}'`. It never reads that cube's column: a dimension whose `sql` is +`{Program}."title"` makes Cube fail the whole model. The member is a declared dimension of that +cube (attribute or time) with no `@via` over the same field when there is one, or the cube's +primary-key dimension for a key field. Otherwise the exporter adds a dimension named after the +field, `public: false`, to that cube. A multi-hop +`@via` (`Week.fkProgram.fkOrg`) joins each hop's cube, and the dimension reads the far cube +(`{Org.name}`); Cube follows the joins. + +### Joins + +Joins exist between cubes only. The exporter never writes a cube so that a reference has +somewhere to go: an `identity.reference` onto an entity that is not a cube makes no join. + +- **A to-one `identity.reference` the cube holds**, onto a cube: a join with `relationship: + many_to_one` and `sql: '{CUBE}."program_id" = {Program}."id"'`, one per reference, in + declaration order. A composite reference joins every column pair, ANDed in the order of the + reference's `@fields`. The key side is the reference's `@references` fields, or else the + referenced entity's `identity.primary`. A reference whose `@fields` and key differ in count is + `ERR_CUBE_UNMAPPABLE_JOIN`, and so is a hop the view's own walk refuses (an ambiguous to-one + relationship). +- **A to-one `relationship.*` whose reference the other entity holds**, onto a cube: a join on + this cube with `relationship: one_to_one` and `sql: '{CUBE}."id" = {Profile}."program_id"'`. +- **Two or more to-one hops onto one entity, or a hop onto the cube's own entity.** Cube allows + one join per target cube and none onto the cube itself. Each such hop gets an **alias cube** + named `_` (`Match_homeRef`, `Match_awayRef`, `Node_fkParent`), and the join and the + dimensions use it. It is never the plain target for one hop and an alias for another, which + would depend on declaration order. An alias cube is a standalone, minimal cube over the + target's table: `public: false`, the primary key, the members the dimensions that reach it + read, and the onward joins of any multi-hop path that continues through it. It does not + `extends` the target, because `extends` copies every measure and rollup of the parent and each + alias would rebuild them. An entity reached only through aliases gets no join-target cube. +- **A cube in a join whose entity has no `identity.primary`** is `ERR_CUBE_NO_PRIMARY_KEY`: Cube + needs a primary key on both sides of a join. +- **A multi-hop `@via` that Cube could join by more than one route** is + `ERR_CUBE_AMBIGUOUS_PATH`, naming both routes. See [Known limits](#known-limits). + +### Measures + +`x` is the `@of` column on `{CUBE}`. A measure's condition is its `@segment`'s filter and its +`@filter`, ANDed, written as one `filters` entry. + +| Measure | Cube | +|---|---| +| `count` | `type: count`, `sql: x` | +| `count` with `@distinct` | `type: count_distinct`, `sql: x` | +| `count` with `@distinct` over a tuple `x1, x2` | Postgres: `type: count_distinct`, `sql: 'ROW(x1, x2)'`, with one `filters` entry `x1 IS NOT NULL AND x2 IS NOT NULL`, so a tuple with a null component is not counted, as in the view. MySQL: the view's own `COUNT(DISTINCT x1, x2)` as a `type: number` measure (a condition `c` makes it `COUNT(DISTINCT CASE WHEN c THEN x1 END, x2)`); MySQL's multi-argument form skips a tuple with a null component and compares by the columns' collation, which a `JSON_ARRAY(x1, x2)` key does not: on mysql:8.4 under the default `utf8mb4_0900_ai_ci` it counted `'abc'` and `'ABC'` as two tuples where the view counts one | +| `sum`, `avg`, `min`, `max` | `type: sum`, `avg`, `min`, `max`, `sql: x` | +| any of the above with a condition `c` | the same, with one `filters` entry `c`. For a Postgres tuple, `c` is ANDed after the not-null terms in that one entry. | +| `measure.ratio` | `type: number`. Postgres: `CAST({num} AS NUMERIC) / NULLIF({den}, 0)`. MySQL: `{num} / NULLIF({den}, 0)`. | +| a `measure.aggregate` with `@default: n` | two members. `Raw` is the aggregate as the rows above write it (its `type`, `sql` and `filters`), with `public: false`. `` is `type: number`, `sql: 'COALESCE({Raw}, n)'`, and carries the measure's `title` and `description`. | +| a `measure.ratio` with `@default: n` | `type: number`, the quotient above inside `COALESCE(…, n)` | + +In a ratio, `{num}` and `{den}` are references to the two operand measures, so each operand is +its full Cube expression, condition included, and an operand need not be listed in any report. +An operand that declares its own `@default` is its `COALESCE` member, so its default reaches the +ratio whether or not the ratio declares one, as in the view (`revenue` with `@default: 0` over +`buyers` reads `0`, not null, for a group with buyers and no revenue). + +`@default` is the value the view reads in place of null (`COALESCE(E, n)`), and Cube reads the +same: a defaulted measure is never null, on an empty group, a group whose rows the measure's +condition filters out, or a ratio whose denominator is zero. Cube has no nullability to declare +for a measure, so nothing else is written for it. A rollup lists ``, never `Raw`. As a +`number` measure, `` is served from a rollup only when the query's dimensions are the +rollup's own, which is the query a report makes. + +No measure sets `format`: Cube's named formats are display hints and the model carries none. + +### Segments and relative dates + +A `segment.filter` becomes a segment whose `sql` is the filter, written by the function that +writes the `WHERE` of a report view (`projection/report-sql.ts` in `codegen-ts`). The exporter +has no second filter translator, so a filter means the same thing in the view and in Cube. + +A relative date `{ "now": "-P7D" }` becomes the view's own SQL, on the Postgres spelling: +`(now() - INTERVAL 'P7D')` for an instant, `((now() AT TIME ZONE 'UTC') - INTERVAL 'P7D')` for +a naive (`@localTime`) timestamp, and `CAST(((now() AT TIME ZONE 'UTC') - INTERVAL 'P7D') AS +DATE)` for a date. The exporter does not map it to a Cube query `dateRange`: segments and +measure filters are SQL, so a relative filter has to be SQL there anyway, and the view's own +SQL is exact. + +### Reports: rollups and scope segments + +A served report writes into the cube of its `@from` entity, a TPH subtype's cube included. The +rollup is named after the report. + +| Report part | In the cube | +|---|---| +| attribute dimensions | `dimensions: [CUBE.d, …]` in listed order. A `@via` dimension is a member of the `@from` cube, so it is listed the same way. | +| one time dimension | `time_dimension: CUBE.d` and `granularity: ` | +| two or more time dimensions, including one dimension at two grains | `time_dimensions:`, a block list with one `{ dimension, granularity }` entry for each, in listed order | +| measures | `measures: [CUBE.m, …]` in listed order | +| `@segment` | `segments: [CUBE.]` | +| `@filter` | a public segment `Scope` on the cube, where `` is the report's name with a lower-case first letter (`RecentRevenue` gives `recentRevenueScope`), whose `sql` is the filter. It is listed after `@segment` in the rollup's `segments`. | +| no dimensions | a rollup of measures only (the totals row) | + +**A report with a relative date gets no rollup.** That is so when its `@filter`, its `@segment`'s +filter, or the condition of any measure it lists (a ratio's operands included) holds one. Cube +builds a rollup when it refreshes it, so the rollup's "now" would be the build's. The view's is +the query's. The scope segment is written whenever the report has a `@filter`, relative date or +not. So a report whose relative date is only in its `@segment` or a measure's condition, and +that has no `@filter`, adds nothing to the cube: that segment and measure are already members of +it. + +The rollup and the scope segment are all the exporter writes for a report. It writes neither a +`refresh_key` nor a partition: Cube's defaults apply, and partitioning is a deployment choice. + +**Rollups are written coarsest first.** Cube answers a query from the first rollup, in +definition order, that can serve it, and a finer rollup can serve a coarser query whose measures +are additive. On Cube 1.7.43, in report order, the query of a totals report (no dimensions) was +answered from the rollup of a report grouped by two dimensions. The rows were right either way, +but the exporter orders each cube's rollups so that a report's query reaches a rollup built for +it before any strictly finer one. The keys, in turn: fewer grouping columns (attribute plus time +dimensions); the coarser time grain, comparing time dimensions in listed order; fewer measures; +report order. + +**The names are one namespace.** A rollup and a `Scope` segment share the cube's member +namespace with its dimensions, measures and segments, because Cube reports a pre-aggregation +named like a member as defined twice. A clash is `ERR_CUBE_MEMBER_COLLISION`. + +**A report with no cube to hold it is refused.** A served report whose `@from` entity is +abstract or has no table is `ERR_CUBE_UNMAPPABLE_REPORT`. The check runs when the run selects at +least one cube. + +In development mode Cube names the table of a rollup `dev_pre_aggregations.__`, +both parts in snake case, joined by two underscores (`week__program_minutes`). + +### Reports with `@spine`: a Cube view + +A served report that declares `@spine` reads its rows from the spine entity: one row for each +distinct dimension tuple among that entity's rows, the ones no fact refers to included, where a +count reads `0` and any other measure null or its `@default` ([reporting](reporting.md)). A rollup +cannot hold those rows. Cube builds a rollup declared on the spine cube from the fact cube, so the +empty rows are missing, and once it exists it changes the answer of a matching query (executed on +Cube 1.7.43: 3 rows where the view has 7). So the report is a Cube **view**, with no rollup and no +scope segment: + +| Part | Written | +|---|---| +| the view | `model/views/.yml`, named after the report, with the report's `title` and `description`. Views and cubes share Cube's one namespace. | +| the spine cube | the cube each listed dimension reaches after the spine's hops: the spine entity's own cube, its join-target cube, or the alias cube of the last hop. It gets one `one_to_many` join onto the report's facts cube (or onto its first chain cube), on the columns of the spine hop's reference. | +| `Facts` | a standalone cube, `public: false`, whose `sql` is `SELECT * FROM ` with the report's `@segment` filter and then its `@filter`, ANDed, as its `WHERE` (none when the report has neither; for a TPH subtype, the base table, its discriminator first). The alias is the one the report view gives `@from`, so the condition reads as the view's join condition does. It holds `@from`'s primary key and `@from`'s own definitions of the measures the report lists, the operands of a listed ratio and the `Raw` of a defaulted measure, in `@from`'s order; what the report does not list is `public: false`. | +| `_` | for a spine of more than one hop, a standalone chain cube, `public: false`, for each entity between `@from` and the spine entity: its table, its primary key and one `one_to_many` join onward, towards the facts. `` is the hop that reaches the entity from `@from`'s side, so `Session.fkWeek.fkProgram` gives `ProgramSessions_fkWeek`. | +| the view's `cubes` | first `join_path: ` (and, for a dimension past the spine, the join path its own joins take from there), including the member each listed dimension reads, under the dimension's name (`id` as `programKey`) and with the dimension's `title`, `description` and `meta.grains`; last, `join_path: [.].Facts`, including the listed measures. | + +The report's scope sits in the facts cube's own `sql`, so it scopes the facts inside the join and +a spine row whose facts are all filtered out keeps its row, as the view's join condition does. A +Cube segment would be a `WHERE` on the whole query, which turns the outer join back into an inner +one. No join is added to `@from`'s own cube or to any cube of an entity between, so every ad-hoc +answer of the ordinary cubes is unchanged: on 1.7.43, `{ "measures": ["Week.weeks"], +"dimensions": ["Program.id"] }` is still rooted at `Week` (3 rows), with `Program`'s joins onto the +facts cubes in place. The measure definitions are copied into the facts cube, which is the cost of +keeping the ordinary cubes as they are. + +A TPH subtype is exported in either place. As the spine entity, its own cube, whose `sql` is the +base table scoped by the subtype's discriminator, is the spine cube, so the view's rows are that +subtype's rows only. As `@from`, the facts cube's `sql` puts the discriminator before the report's +scope. Both are correct by construction, and the report view lowering refuses both reports (a TPH +subtype has no table of its own), so there is no view to compare them with. + +A facts or chain cube is private plumbing: it carries no `title` or `description`, while the view +carries the report's. + +Executed on Cube 1.7.43 before this was built, and checked by the `cube` lane since: a view +includes a private primary key and a `public: false` member under an alias, and one member twice +under two aliases (two dimensions over one field and path); the include-level +`title`, `description` and `meta` reach `/v1/meta`; the roster view returns every program, with +`weeks` `0`, its plain sum null and its defaulted measures `0` for the programs with no weeks; the +scoped facts cube keeps the programs whose weeks are all short; and a two-hop chain returns the +view lowering's rows. A view is answered from the tables. + +### Names, quoting and escaping + +| Rule | Behaviour | +|---|---| +| cube name | the entity's name. Two entities of one name in two packages, or an alias, facts or chain cube or a `@spine` report's view named like a cube, is `ERR_CUBE_NAME_COLLISION`, naming both (the message says "cube or view" when one is a view): Cube's cubes and views share one namespace. Rename one, or narrow the generator's `filter` to leave one out; the filter helps only when no `@via` reaches the entity it leaves out, since an entity a `@via` reaches is still written as a join-target cube, unless every hop onto it goes through an alias cube. | +| member name | the dimension, measure or segment name as written, so a report field and its Cube member share a name | +| a name Cube refuses | Cube names start with a letter, hold only letters, digits and `_`, and are not a Python keyword (`from`, `class`, `in`, `is`, `not`, `and`, `or`, `if`, `else`, `for`, `while`, `with`, `as`, `def`, `return`, `yield`, `import`, `pass`, `global`, `nonlocal`, `lambda`, `del`, `assert`, `break`, `continue`, `try`, `except`, `finally`, `raise`, `async`, `await`, `True`, `False`, `None`, `elif`). That is `ERR_CUBE_INVALID_NAME`, naming the node. The exporter never renames: the name is the report field's. | +| members the exporter adds | primary-key dimensions, reached-column dimensions, the `Raw` measure of a defaulted measure, `Scope` segments, rollups. A name that collides with another member of the cube is `ERR_CUBE_MEMBER_COLLISION`, naming both. The one exception is a declared dimension over a key field under its own name, which is that key dimension (see [Cubes and primary keys](#cubes-and-primary-keys)). | +| identifiers | every table, schema and column is quoted: `"…"` on Postgres, backticks on MySQL | +| string literals | SQL quoting first (`'` doubled; MySQL also doubles `\`). Cube compiles every `sql` as a template literal, where a backslash is an escape (`\b` a backspace, `\_` a plain `_`, a trailing `\` swallows the closing quote) and `{x}` a member reference. So every `\` is then doubled, and after that `{` becomes `\{` and `}` becomes `\}` (in that order, so a backslash before a brace stays a backslash). Then, when the literal holds a Jinja opener (`{{`, `{%` or `{#`), it is wrapped in `{% raw %}…{% endraw %}` for Jinja, because a backslash does not stop Jinja. A wrapped literal holding `endraw` is `ERR_CUBE_UNESCAPABLE_LITERAL`; one that is not wrapped is never inside a raw block, so `endraw` there is plain text. A column name written with `@column` gets the same treatment. The `cube` lane reads each literal of the `escaping` case back from Cube's `/v1/sql`, where it is the view's own SQL. | +| free text | `title` and `description`. Cube reads them as templates too: `{x}` is a member reference, `${x}` an interpolation and a backslash an escape. Each `\` is doubled, each brace escaped, the text raw-wrapped when the original holds `{{`, `{%` or `{#` (the same rule as a SQL literal), and the result is written as a JSON string. Raw-wrapped text holding `endraw` is `ERR_CUBE_UNESCAPABLE_LITERAL`. | +| YAML scalars | A `sql`, `sql_table` or filter value is single-quoted (`'` doubled), or written as a JSON double-quoted string when it holds a line break, a control character, U+007F to U+009F, U+2028, U+2029 or U+FEFF. Names, types and `CUBE.` references are plain, except a name a YAML reader would take for a boolean or null (`true`, `false`, `null`, `yes`, `no`, `on`, `off`, `y`, `n`, in any case), which is single-quoted. The emitter is hand-written, with no YAML dependency, so the bytes are the same on every run: two-space indent, LF line endings, one trailing newline, no trailing spaces. | + +### What the exporter refuses + +Anything this mapping cannot express is a generation error that names the node and says what +to change. Nothing is dropped silently. The `ERR_CUBE_*` codes belong to this generator: they +are printed in its messages and are not loader codes. + +| Code | Raised when | +|---|---| +| `ERR_CUBE_UNMAPPABLE_DIMENSION` | a dimension reads an array, object or map field, or a field with `@objectRef`; or its `@via` crosses a to-many hop, comes back to an entity already on the path (a self-referencing hop is the exception), reaches an entity with no table, or passes through one cube twice | +| `ERR_CUBE_UNMAPPABLE_JOIN` | a reference's `@fields` do not pair with the key it references, or the view's walk refuses a hop | +| `ERR_CUBE_UNMAPPABLE_REPORT` | a served report's `@from` has no cube to hold its rollup | +| `ERR_CUBE_AMBIGUOUS_PATH` | a multi-hop `@via` reads a cube that the cube graph reaches by more than one path | +| `ERR_CUBE_NO_PRIMARY_KEY` | a cube in a join has no `identity.primary` | +| `ERR_CUBE_INVALID_NAME` | a cube or member name Cube cannot take | +| `ERR_CUBE_MEMBER_COLLISION` | two members of one cube would have one name | +| `ERR_CUBE_NAME_COLLISION` | two cubes, or a cube and a view, would have one name | +| `ERR_CUBE_UNESCAPABLE_LITERAL` | a literal, an identifier or free text that is raw-wrapped (it holds a Jinja opener) holds `endraw` | +| `ERR_CUBE_UNSUPPORTED_DIALECT` | the dialect is neither `postgres` nor `mysql` and the run would write a cube | + +## Which files a run writes + +What a cube holds never depends on the run. The build covers the whole loaded model, narrowed +only by the generator's own `filter`, which is fixed config. So `meta gen Week` and a full run +write the same bytes for `Week.yml`. + +The run's selection decides which files are written. That selection is `meta gen `, or +the `scope` of the project's [collection](metadata-sources.md). The run writes the selected +entities' own cubes and every cube they reach through joins, transitively. It writes a reached +cube because a selected cube changes it: `Week.programTitle` adds `title` to `Program`, and a +`Program.yml` left out might lack that member. A report is never selected itself; its rollup +rides with its `@from` cube. A `@spine` report's view is written when every cube its join paths +name is written: it reads them and nothing else. A run that selects no entity with reporting +vocabulary writes nothing and raises nothing, under any dialect. + +`scope` narrows what is written, not what is built. An error on an entity outside the scope +still fails the run. To step around an entity, exclude it with the generator's `filter`. + +**Cleanup.** Cube compiles the whole model directory, so a stale `.yml` left by a removed +or renamed entity, or by a join no longer reached, can break the compile for every cube. The +generator opts in to the runner's orphan cleanup for its own files, the direct `.yml` children +of `model/cubes/` and `model/views/` under its target. The runner removes such a file only when a previous run +wrote it, this run did not write it again, and it is byte-identical to what was written. A file +you edited is refused and named, never deleted, and a file the generator never wrote is never a +candidate. A run that names entities (`meta gen `) skips the cleanup. A project `scope` +is persisted config and not a narrowed run, so after you change it, a full run removes the +untouched cube files the new scope no longer selects. `meta gen --dry-run` reports the pending +removals and performs none. + +## Querying a report in Cube + +This is the Cube query that reproduces a served report. The live check builds it from the report. + +```json +{ "measures": [".", "…"], + "dimensions": [".", "…"], + "timeDimensions": [{ "dimension": ".