Conversation
New export-only spoke `converters/bigquery/` mapping an Apache Ossie semantic model to a BigQuery native property graph with graph measures: - datasets -> NODE TABLES (KEY from primary_key, PROPERTIES from fields) - relationships -> EDGE TABLES (SOURCE/DESTINATION KEY ... REFERENCES) - single-table metrics -> inline MEASURE(<agg>) AS <name> - description + ai_context.synonyms -> OPTIONS(description=...) A graph measure binds an aggregate to exactly one table's KEY, so results stay correct under fan-out; the cross-table rollup happens at query time via GRAPH_EXPAND + AGG. Metrics that span multiple datasets (or none) cannot be a single MEASURE and are skipped with a warning, deferring to native BigQuery measure features as they expand. Pure str->str transform, PyYAML-only runtime dependency; no BigQuery connection or deploy. Import (BQ DDL -> Ossie) is a stub for now, consistent with the snowflake/polaris export-only spokes. Includes a golden-file test over a trimmed TPC-DS star plus unit tests for measure placement/skip, edge keys, single-root validation, dialect fallback, and description folding, and a CI workflow mirroring the databricks spoke.
Rewrite converters/bigquery/README.md as a task-oriented guide: quick start with a complete copy-pasteable model and its exact emitted DDL, CLI reference, Python API, per-rule mapping examples (dataset/relationship/metric) drawn from the TPC-DS model, a metrics-and-measures section, deployment via bq, and a querying section with correct GRAPH_EXPAND + AGG consumption queries (measures wrapped in AGG, <Label>_<property> column naming, cache caveat, BQ.SHOW_GRAPH_EXPAND_SCHEMA), plus requirements, limitations, and a warnings reference table. All DDL and query examples verified against real converter output and the BigQuery graph-measures docs.
Re-indent the "How the model maps" DDL excerpts to match the converter's actual output exactly (node/edge entries nested two spaces inside NODE TABLES/EDGE TABLES; the measure shown in its PROPERTIES context), and frame them as excerpts of the single CREATE OR REPLACE PROPERTY GRAPH statement. Expand the export-only limitation to explain the rationale (Ossie is the authoring source of truth; the DDL is a deliberately lossy generated artifact, so it is not a faithful round-trip source) rather than just noting import is unimplemented.
Align the emitted CREATE OR REPLACE PROPERTY GRAPH layout with the style BigQuery uses in its own graph-measures documentation: - Indent NODE TABLES / EDGE TABLES / OPTIONS one level under CREATE (element tables and their clauses nest from there), via a _table_group helper so the nesting stays depth-driven. - Write SOURCE KEY (...), DESTINATION KEY (...), and REFERENCES <table> (...) with a space before the paren, while KEY(...)/PROPERTIES(...)/MEASURE(...)/ OPTIONS(...) stay attached. Regenerate the golden fixture, update edge-clause test assertions, and bring every README DDL example in line with the new output.
Validated the emitted DDL end-to-end against a real BigQuery instance (deploy + GRAPH_EXPAND/AGG measure queries, confirming measures are fan-out-safe). Two cases the offline tests missed: - Hyphenated GCP project IDs (e.g. `my-proj.ds.tbl`) were emitted unquoted and flagged as non-identifiers, producing invalid SQL. Table qualifier validation now permits hyphens in a dotted reference, so the whole path is backtick-wrapped. - A MEASURE can only aggregate columns exposed as node properties; a metric over a column no field declared produced DDL BigQuery rejects. The emitter now auto-exposes any measure-referenced column that isn't already a property (new _expr.referenced_columns helper). Add regression tests for both and document the auto-exposure behavior. The shipped TPC-DS golden model also deploys cleanly to live BigQuery; golden output is unchanged.
…ming The BQ Graph DDL -> Ossie direction was only a not-implemented stub. Since import is deferred with no concrete use case, drop the stub module, its CLI `import` subcommand, and the package export rather than ship dead code. The converter is now cleanly export-only, matching the snowflake/polaris spokes. Also standardize prose on the approved brand name "BigQuery Graph" (capital G) across the README, docstrings, comments, and package metadata.
The aggregate feature is just "measures". Update prose to "BigQuery Graph with measures" across the READMEs, docstrings, comments, and package metadata. The real doc URL (docs/graph-measures) is left as-is.
The docstrings had overlong summary lines that forced a blank-line split with dangling continuation fragments, plus several mid-phrase wraps. Rewrite them as proper PEP-257 docstrings: a short complete summary line, then cleanly wrapped bodies, all within 80 columns. No behavior change.
The `_common.py`/`_expr.py` split was inherited from the bidirectional
databricks spoke, but here both files were consumed only by the emitter and
`_common.py` had degenerated into a junk drawer (error type, validation, YAML
loading, and Ossie accessors under a vague name), plus two dead symbols
(`is_simple_identifier`, `VENDOR`).
Reorganize into two clearly-named modules:
- converter.py (was ossie_to_bq_graph.py) -- the emitter, plus
ConversionError and the input loading/validation/accessor
helpers that only it uses
- sql_expressions.py (was _expr.py) -- the literal-aware SQL scanning, the one
genuinely reusable, self-contained concern
Delete _common.py and its dead code; rename the test file to match. No
behavior change; 40/40 tests pass.
sql_expressions.py hand-rolled SQL analysis with regex, using two different string-literal strategies for the same concern and a hand-maintained keyword/type blocklist. Rewrite the three helpers (referenced_datasets, strip_qualifier, referenced_columns) on top of sqlglot's parser, reading facts from the expression's column nodes. This is the house style already used by the dbt and gsf spokes, and lets the grammar handle string literals, function names, keywords, and type names instead of special cases. Emitted DDL is unchanged (sqlglot preserves the expressions verbatim in these cases, including the `||` operator). Also drop the sql_body() test helper: the golden .sql fixture no longer carries an ASF header, so the export test compares byte-for-byte against the raw fixture. Fixtures are test data and the repo enforces no license header on them.
| return f'"{escaped}"' | ||
|
|
||
|
|
||
| def _describe(description, synonyms): |
There was a problem hiding this comment.
This is off. In BQ Graph, you have OPTIONS. OPTIONS contain synonyms. Please do a thorough job to understand what's possible. Do not put a whole bunch of hacks like this.
| return "\n\n".join(parts) if parts else None | ||
|
|
||
|
|
||
| def _options_clause(description, synonyms): |
There was a problem hiding this comment.
Such a short function, let's inline it.
|
|
||
| # Apache Ossie spec version this converter targets (see core-spec/). | ||
| # | ||
| # NOTE: this is an exact-match check. Like the databricks spoke, this converter |
There was a problem hiding this comment.
Do not mention the names of other products other than BigQuery in your folder. For example, Databricks is mentioned here; please do not do this.
| datasets = {} | ||
| for ds in dataset_list: | ||
| name = require_str(ds, "name", f"Model '{model_name}': dataset") | ||
| if name in datasets: |
There was a problem hiding this comment.
This is essentially validating the model. Are there already existing validation modules in Ossie codebase? We need to make sure our converter is reusing the rest of the codebase as much as possible.
| def _place_metric(model_name, metric, datasets, measures_by_dataset): | ||
| name = require_str(metric, "name", f"Model '{model_name}': metric") | ||
| expr = pick_expression(metric.get("expression"), f"metric '{name}'") | ||
| if expr is None: |
There was a problem hiding this comment.
Does it have to be BigQuery or ANSI SQL dialect? If it's Databricks, for example, is it possible to have such an input? If so, shall we convert that into BigQuery dialect using SQLGlot?
| # specific language governing permissions and limitations | ||
| # under the License. | ||
|
|
||
| """Dataset-qualifier analysis for the SQL expressions in a semantic model. |
There was a problem hiding this comment.
This file is so short. Shall we consider inlining it to the converter.py?
| # under the License. | ||
|
|
||
| """Shared test helpers: fixture loading.""" | ||
|
|
There was a problem hiding this comment.
This file is so short. Can we inline it? What's the point of having a separate file?
…ialect transpile - Emit description and synonyms as native BigQuery Graph OPTIONS (`description="..."`, `synonyms=[...]`) on the element DEFAULT LABEL and on properties, instead of folding synonyms into free-text descriptions. Verified the syntax (incl. `synonyms` array and `DEFAULT LABEL` placement) against live BigQuery. - Delegate model parsing and structural validation to the core apache-ossie pydantic models (OSIDocument.model_validate), reusing the shared schema rather than re-implementing require/require_str/version handling. - Transpile non-BigQuery/ANSI SQL dialects (e.g. Snowflake, Databricks) to BigQuery with sqlglot instead of skipping them; only non-SQL dialects (MDX/TABLEAU/MAQL) or a missing expression are skipped. - Inline the small sql_expressions module into converter.py and the test _util helper into the test module. - Drop other-product references from the converter folder. All 43 tests pass; the regenerated golden DDL is accepted by live BigQuery.
| # Ossie SQL dialects mapped to their sqlglot names. BigQuery and ANSI_SQL are | ||
| # used verbatim; the rest are transpiled from these names to BigQuery. Non-SQL | ||
| # dialects (MDX, TABLEAU, MAQL) are absent -- they have no BigQuery rendering. | ||
| _SQLGLOT_DIALECT = { |
There was a problem hiding this comment.
This is not future compatible because there will be more dialects added. In that case, we will have a stale list. How do we do it in a more future-proof way?
| return _INDENT * depth + text | ||
|
|
||
|
|
||
| def _table_group(keyword, entries): |
There was a problem hiding this comment.
The function names are very ambiguous. What do you mean by a table group? In your comment, you said "render clauses and their entries". They don't sound like table group to me. Please make sure all function names are clear.
…learer names - Resolve transpilable dialects dynamically via sqlglot's registry instead of a hardcoded SNOWFLAKE/DATABRICKS table and dialect-preference tuple, so a SQL dialect added to the core spec is picked up with no code change and a non-SQL dialect is skipped automatically. - Rename ambiguous layout helpers (_table_group -> _render_tables_clause, _properties_block -> _render_properties_clause) and tighten their docstrings. - Scrub remaining non-BigQuery product names from docstrings and README, and drop the stale hardcoded dialect lists there.
| return "\n".join(lines) | ||
|
|
||
|
|
||
| def _render_field_property(dataset, field): |
There was a problem hiding this comment.
For each function, let’s have a proper comment to describe what it does.
| return "\n".join(lines) | ||
|
|
||
|
|
||
| def _render_field_property(dataset, field): |
There was a problem hiding this comment.
What does a field property mean? I have not heard that term before.
| return f'"{escaped}"' | ||
|
|
||
|
|
||
| def _options_clause(description, synonyms): |
There was a problem hiding this comment.
This function name is inconsistent with other function names we commented before. So let's really make sure to go over all the function names in the converter and make sure they are clearly reflecting what they do, not only for this function, but for all functions in the converter. We really have to write clean code. This is not high enough quality.
| return f"DEFAULT LABEL {opts}" if opts else None | ||
|
|
||
|
|
||
| def _description_of(obj): |
There was a problem hiding this comment.
By looking at this function name, I don’t know what it does. Is it a description of what? What is the input type? This is very confusing. You can do much better.
| return None | ||
|
|
||
|
|
||
| def _referenced_datasets(expression, dataset_names): |
There was a problem hiding this comment.
The function name doesn't really reflect what the comment says of what it does. I find the code hard to read. I care about the code quality. Make sure the function names are really readable and really reflect what it does. Please do a better job in code quality.
| return name | ||
|
|
||
|
|
||
| def _transpile(sql, read_dialect, what): |
There was a problem hiding this comment.
The parameter of what? It’s really hard to understand what that parameter means and how it’s used.
…ction - Give every function a docstring stating what it does, its inputs, and its return, per review. - Rename functions so the name reflects the action and is consistent across the module: the DDL-emitting helpers are all _render_* (_options_clause -> _render_options_clause, _label_options -> _render_default_label_clause, _string_literal -> _render_string_literal), _convert_model -> _render_property_graph, _render_field_property -> _render_property_from_field (drops the unclear "field property" term), _validate -> _parse_document, _place_metric -> _register_metric_measure, _referenced_datasets/_referenced_columns -> _find_referenced_*, _strip_qualifier -> _strip_dataset_qualifier, _parse -> _parse_sql, _validate_single_root -> _warn_unless_single_root, _line -> _indented_line, _qualify_graph/_qualify_table -> _qualify_graph_name/_qualify_table_name, _description_of/_synonyms_of -> _element_description/_element_synonyms. - Rename the opaque 'what' parameter to 'context_label' (and 'key' to 'field_name', 'obj' to 'element', 'valid' to 'node_names') and document what each means. Pure rename/documentation pass; output DDL is byte-identical (golden test unchanged) and all 43 tests pass.
BigQuery edge tables can carry PROPERTIES just like node tables. The core spec has no field slot on a relationship yet, so edge properties are declared in a relationship custom_extensions entry whose JSON payload mirrors the core `fields` shape verbatim -- the form a spec-native relationships[].fields would take, promotable later with no change to authored models. Edge fields validate with the shared OSIField model and render through the same path as node fields (bare column or <expr> AS <name>, optional inline OPTIONS). Adds a readable golden fixture pair (edge_properties_ossie.yaml -> edge_properties_graph.sql) plus unit tests; full suite 52 passing.
Give every metadata-bearing object in both golden fixtures a description and a structured ai_context (synonyms): the model, every node, every property, both edges, each measure, and the edge properties. This showcases how Ossie's semantic AI context flows onto BigQuery's native graph OPTIONS(description=..., synonyms=[...]) at every level, edge properties included. Goldens regenerated; full suite 52 passing.
The edge-properties custom extension is a BigQuery-specific convention until the core spec gains an edge-field slot, so it belongs to the GOOGLE vendor rather than the vendor-neutral COMMON namespace. Make the tag meaningful: the converter now reads edge properties only from the GOOGLE-owned extension and leaves any other vendor's extension untouched. Full suite 53 passing.
The hand-authored edge-properties example is a property graph, so drop the data-warehouse framing: rename the fixture to orders_ossie/orders_graph and describe nodes as entities (an order, a customer) rather than fact/dimension tables. Both graphs target BigQuery, so the fixtures and test helpers now declare the BIGQUERY expression dialect instead of ANSI_SQL (the BIGQUERY-over-ANSI preference test keeps its explicit dialects). Output is unchanged for tpcds; the orders golden is regenerated for the new descriptions.
Validated the generated DDL against live BigQuery. BigQuery treats a graph label or property name as an implicit synonym of itself and rejects a CREATE PROPERTY GRAPH whose synonyms list repeats that name, or lists the same synonym twice -- both compared case-insensitively. The converter now drops such synonyms (keeping the first spelling) and warns, so the emitted DDL is always accepted. Also fixes the orders fixture, whose 'orders' synonym collided with its label. Verified live: both goldens and a collision-laden model now deploy cleanly, edge properties are queryable, and measures aggregate once per key under fan-out.
A plain foreign-key relationship is many-to-one: the edge is backed by the `from` table and each `from` row links to at most one `to` row. A many-to-many link (a student takes many courses; a course holds many students) needs a junction table, which BigQuery models as an edge whose SOURCE and DESTINATION keys both sit on the junction and REFERENCE two different nodes. Add an additive `association` block, carried in the same GOOGLE custom_extensions payload that already holds edge properties: it names the junction table and its two foreign-key column sets, while the relationship's from_columns/to_columns stay the referenced key columns on the endpoint nodes. The direct foreign-key path is unchanged. GRAPH_EXPAND only walks many-to-one / one-to-one edges, so a many-to-many edge is consumed with MATCH, carries no measures, and is excluded from the single-root check. Verified against live BigQuery: the generated golden deploys and traverses in both directions with its edge properties. Adds the enrollments_graph golden fixture and unit tests.
Replace the relationship-attached `association` block with a first-class
edge: an edge is modeled exactly like a dataset (its own `source`,
`primary_key`, and `fields`) plus two endpoints (`source_key`/
`destination_key`), each `{columns, node, references}` reading left-to-right
as its SOURCE/DESTINATION KEY ... REFERENCES clause. This mirrors BigQuery,
where an EDGE TABLE is a NODE TABLE plus two endpoints, and drops the
confusing overload where a relationship's from_columns/to_columns changed
meaning between the FK and junction cases.
Core-spec `relationships` stay as sugar for a many-to-one FK edge; both
relationships and first-class `edges` normalize to one internal shape and
render through a single path, so the tpcds/orders goldens are byte-identical.
First-class edges ride in a model-level GOOGLE custom_extensions `edges` list,
shaped as a future spec-native `edges:` (a sibling of `datasets:`) would be.
- rename fixtures mn_ossie.yaml/mn_graph.sql -> enrollment_ossie.yaml/
enrollment_graph.sql; regenerate the golden (warning-free)
- rewrite the M:N tests for the first-class edge shape (endpoints, key/
references defaults, arity + structure validation, root-check suppression)
- README: rewrite the Many-to-many edges section for first-class edges
- live-validated on BigQuery: CREATE + forward/reverse MATCH + edge property
+ genuine M:N (course held by 3 students; students taking 2 courses each)
Every edge is a `relationship`, whatever its cardinality, so a many-to-many link now lives in the same `relationships:` block as any many-to-one edge instead of a separate model-level `edges` list. The native relationship fields name the two node endpoints (`from`/`to` are the nodes; `from_columns`/`to_columns` are the columns referenced on them). The one thing the core spec cannot express yet -- the through-table -- rides in a GOOGLE-owned `relationship` custom extension on that relationship: its `source`, `primary_key`, and `source_key`/`destination_key` join columns. `from` renders as BigQuery's SOURCE and `to` as its DESTINATION. The through-table appears only as `source`; it is never a dataset, since a dataset is an entity (a node), not a relationship. Edge properties (1:M and M:N) unify under `relationship.fields`. Both cardinalities normalize to one internal shape rendered by one path, so the tpcds/orders golden outputs stay byte-identical. A relationship has no `description` field, so its human description is authored as `ai_context.instructions` and rendered on the edge label; descriptions and AI context are set on every object in the enrollment fixture. Live-validated on BigQuery (CREATE + M:N MATCH).
Fixes five robustness gaps found in code review: - Reject M:N intent without a through-table: a relationship extension carrying source_key/destination_key/primary_key but no `source` now raises instead of silently emitting a wrong one-to-many FK edge. - Validate relationship (edge) names are unique and distinct from dataset (node) names, mirroring the existing dataset-name check; duplicates would emit DDL BigQuery rejects at deploy. - A non-list `fields` in the extension raises a clean ConversionError instead of an uncaught TypeError. - Merge multiple Google-owned `relationship` extensions loss-free: `fields` lists concatenate, and any other key redefined with a conflicting value raises rather than being silently overwritten. - Warn when an edge endpoint REFERENCES columns that are not the referenced node's KEY (a deploy-time constraint, like the single-root check). Adds regression tests for each and documents the new requirements and warning.
Add a per-key legend above the GOOGLE relationship custom_extensions in the enrollment (many-to-many) and orders (one-to-many with edge properties) fixtures, explaining source/primary_key/source_key/ destination_key/fields and noting why the comments must sit outside the data: | block.
| synonyms: | ||
| - takes | ||
| - registered for | ||
| # The through-table rides here, in a GOOGLE-vendored `relationship` |
There was a problem hiding this comment.
Let's just call it edge table.
There was a problem hiding this comment.
Done in 8a30779 — switched the header and the extension legend from "through-table"/"junction" to BigQuery's "edge table" term (the association table is declared inside EDGE TABLES).
| # Its presence is what marks this edge many-to-many; | ||
| # a relationship with no `source` is a plain 1:M FK. | ||
| # primary_key -- the junction's own key, used to de-duplicate rows. | ||
| # source_key -- junction column(s) that join to the `from` node |
There was a problem hiding this comment.
Similarly here, edge table.
There was a problem hiding this comment.
Done in 8a30779 — same rename here; source_key/destination_key/primary_key are now described as the edge table's own columns/key.
| # untouched. The converter reads these keys: | ||
| # source -- the junction/through-table (campus.public.enrollment). | ||
| # Its presence is what marks this edge many-to-many; | ||
| # a relationship with no `source` is a plain 1:M FK. |
There was a problem hiding this comment.
If a relationship in this extension doesn't have a source, does it mean it's not an M to N? If it's not M to N, should it just fall back to the oasis original spec without using this extension? Please be very clear.
Do we have a test case covering this case? You have an extension for relationships but do not have a source specified.
There was a problem hiding this comment.
Clarified in 8a30779. Yes: no source means it is not many-to-many. The relationship then stays an ordinary one-to-many FK edge (from_columns -> to_columns), and the extension may still legitimately carry only fields (edge properties) — that is exactly the orders fixture. It is not a full fallback that ignores the extension; the extension is still read for fields. Supplying the M:N-only keys (source_key/destination_key/primary_key) without a source is a hard error (ConversionError), not a silent 1:M fallback, so ambiguous intent fails loudly.
Test coverage:
test_extension_without_source_is_one_to_many(new in 8a30779) — extension with onlyfields, nosource-> asserts the 1:M edge shape (SOURCE KEY (order_id) REFERENCES orders (order_id)).test_many_to_many_keys_without_source_raise— M:N keys but nosource-> raises.- The orders golden also covers the no-source, fields-only form end to end.
…fixture Address review on enrollment_ossie.yaml: - Use BigQuery's "edge table" term for the many-to-many backing table instead of "through-table"/"junction" throughout the header and the custom-extension legend. - State explicitly that a relationship whose GOOGLE extension has no `source` is NOT many-to-many: it stays a one-to-many FK edge (the extension may then carry only edge `fields`), and that the M:N-only keys without a `source` are an error, not a silent fallback. - Add test_extension_without_source_is_one_to_many asserting the 1:M edge shape for a fields-only (no-source) extension.
Review statusLatest batch (enrollment fixture) — addressed in
Earlier batch (converter code, resolved before this push):
All 88 tests pass. |
Reorganize the README so a first-time reader can follow one train of thought: why -> install -> see it work end to end -> understand the mapping -> run it in BigQuery -> reference -> development. - Quick start now walks the full author -> convert -> deploy -> query loop as numbered steps, instead of dropping the reader into CLI/API reference right after their first conversion. - Group pure look-up material (CLI, Python API, requirements, limitations, warnings) under one Reference section at the end. - Fold the separate "Metrics and measures" section into the "Metric -> measure" mapping rule so all mapping rules live together. - Merge "Deploying" and "Querying" into one "Run it in BigQuery". - Align the many-to-many prose with the fixture's "edge table" terminology and state explicitly that no source means not many-to-many. No SQL/YAML examples or technical content changed.
Summary
Adds a new export-only converter spoke,
converters/bigquery/, mapping an Apache Ossie semantic model to a BigQuery native property graph with graph measures (docs). No BigQuery connection or deploy — it emits a singleCREATE OR REPLACE PROPERTY GRAPHDDL statement.Mapping
dataset(source=project.dataset.table)KEYfromprimary_key,PROPERTIESfromfields)relationshipSOURCE KEY … REFERENCES/DESTINATION KEY … REFERENCES)metricMEASURE(<agg>) AS <name>on the owning nodefield<expr> AS <name>)description+ai_context.synonymsOPTIONS(description="…")A graph measure binds an aggregate to exactly one table's
KEY, so results stay correct under fan-out joins; the cross-table rollup happens at query time viaGRAPH_EXPAND + AGG. This maps Ossie metrics straight onto the native measure feature rather than reinventing decomposition/recombination.Design notes
import(BQ DDL → Ossie) is a stub, consistent with thesnowflake/polarisspokes.MEASURE()only. A metric that spans multiple datasets (or none) cannot be a single measure and is skipped with a warning — deferring to native BigQuery measure features as they expand, not a custom workaround.str -> strtransform; PyYAML is the only runtime dependency. No core-spec change (BIGQUERYis already a first-class dialect).Tests
tpcds_ossie.yaml→tpcds_graph.sql).primary_keynode drop + dangling-edge drop, single-root validation (0/1/>1 roots),BIGQUERY→ANSI_SQLdialect fallback, computed-vs-simple field emission, and description/synonym folding.examples/tpcds_semantic_model.yaml: 5 node tables, 4 edges, 3 measures onstore_sales; the two composite metrics are skipped with warnings.Follow-ups
.github/workflows/converter-bigquery-ci.yml, mirroring the databricks spoke) is ready but not in this push — the current token lacks GitHubworkflowscope. It can be added in a follow-up push once scope is granted.importdirection (BQ property-graph DDL → Ossie).