Skip to content

Add BigQuery property-graph (Graph + Measures) converter - #1

Draft
libei wants to merge 25 commits into
mainfrom
feat/bigquery-converter
Draft

libei wants to merge 25 commits into
mainfrom
feat/bigquery-converter

Conversation

@libei

@libei libei commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a new export-only converter spoke, converters/bigquery/, mapping an Apache Ossie semantic model to a BigQuery native property graph with graph measures (docs). No BigQuery connection or deploy — it emits a single CREATE OR REPLACE PROPERTY GRAPH DDL statement.

Mapping

Apache Ossie BigQuery property graph
dataset (source = project.dataset.table) NODE TABLE (KEY from primary_key, PROPERTIES from fields)
relationship EDGE TABLE (SOURCE KEY … REFERENCES / DESTINATION KEY … REFERENCES)
single-table metric inline MEASURE(<agg>) AS <name> on the owning node
field graph property (bare column, or <expr> AS <name>)
description + ai_context.synonyms OPTIONS(description="…")

A graph measure binds an aggregate to exactly one table's KEY, so results stay correct under fan-out joins; the cross-table rollup happens at query time via GRAPH_EXPAND + AGG. This maps Ossie metrics straight onto the native measure feature rather than reinventing decomposition/recombination.

Design notes

  • Export only for now; import (BQ DDL → Ossie) is a stub, consistent with the snowflake/polaris spokes.
  • Native MEASURE() only. A metric that spans multiple datasets (or none) cannot be a single measure and is skipped with a warning — deferring to native BigQuery measure features as they expand, not a custom workaround.
  • Pure str -> str transform; PyYAML is the only runtime dependency. No core-spec change (BIGQUERY is already a first-class dialect).
  • BigQuery requires exactly one root node table; a multi-root/cyclic graph emits a warning (not an error).

Tests

  • Golden-file export over a trimmed TPC-DS star (tpcds_ossie.yamltpcds_graph.sql).
  • Unit tests for measure placement/skip, unsupported-aggregate handling, edge keys & column ordering, missing-primary_key node drop + dangling-edge drop, single-root validation (0/1/>1 roots), BIGQUERYANSI_SQL dialect fallback, computed-vs-simple field emission, and description/synonym folding.
  • CLI smoke on examples/tpcds_semantic_model.yaml: 5 node tables, 4 edges, 3 measures on store_sales; the two composite metrics are skipped with warnings.

Follow-ups

  • The CI workflow (.github/workflows/converter-bigquery-ci.yml, mirroring the databricks spoke) is ready but not in this push — the current token lacks GitHub workflow scope. It can be added in a follow-up push once scope is granted.
  • Live validation of the emitted DDL against a real BigQuery dataset.
  • The import direction (BQ property-graph DDL → Ossie).

libei added 10 commits August 8, 2026 01:47
New export-only spoke `converters/bigquery/` mapping an Apache Ossie
semantic model to a BigQuery native property graph with graph measures:

- datasets      -> NODE TABLES (KEY from primary_key, PROPERTIES from fields)
- relationships -> EDGE TABLES (SOURCE/DESTINATION KEY ... REFERENCES)
- single-table metrics -> inline MEASURE(<agg>) AS <name>
- description + ai_context.synonyms -> OPTIONS(description=...)

A graph measure binds an aggregate to exactly one table's KEY, so results
stay correct under fan-out; the cross-table rollup happens at query time via
GRAPH_EXPAND + AGG. Metrics that span multiple datasets (or none) cannot be a
single MEASURE and are skipped with a warning, deferring to native BigQuery
measure features as they expand. Pure str->str transform, PyYAML-only runtime
dependency; no BigQuery connection or deploy. Import (BQ DDL -> Ossie) is a
stub for now, consistent with the snowflake/polaris export-only spokes.

Includes a golden-file test over a trimmed TPC-DS star plus unit tests for
measure placement/skip, edge keys, single-root validation, dialect fallback,
and description folding, and a CI workflow mirroring the databricks spoke.
Rewrite converters/bigquery/README.md as a task-oriented guide: quick start
with a complete copy-pasteable model and its exact emitted DDL, CLI reference,
Python API, per-rule mapping examples (dataset/relationship/metric) drawn from
the TPC-DS model, a metrics-and-measures section, deployment via bq, and a
querying section with correct GRAPH_EXPAND + AGG consumption queries (measures
wrapped in AGG, <Label>_<property> column naming, cache caveat,
BQ.SHOW_GRAPH_EXPAND_SCHEMA), plus requirements, limitations, and a warnings
reference table. All DDL and query examples verified against real converter
output and the BigQuery graph-measures docs.
Re-indent the "How the model maps" DDL excerpts to match the converter's
actual output exactly (node/edge entries nested two spaces inside NODE
TABLES/EDGE TABLES; the measure shown in its PROPERTIES context), and frame
them as excerpts of the single CREATE OR REPLACE PROPERTY GRAPH statement.

Expand the export-only limitation to explain the rationale (Ossie is the
authoring source of truth; the DDL is a deliberately lossy generated
artifact, so it is not a faithful round-trip source) rather than just noting
import is unimplemented.
Align the emitted CREATE OR REPLACE PROPERTY GRAPH layout with the style
BigQuery uses in its own graph-measures documentation:

- Indent NODE TABLES / EDGE TABLES / OPTIONS one level under CREATE (element
  tables and their clauses nest from there), via a _table_group helper so the
  nesting stays depth-driven.
- Write SOURCE KEY (...), DESTINATION KEY (...), and REFERENCES <table> (...)
  with a space before the paren, while KEY(...)/PROPERTIES(...)/MEASURE(...)/
  OPTIONS(...) stay attached.

Regenerate the golden fixture, update edge-clause test assertions, and bring
every README DDL example in line with the new output.
Validated the emitted DDL end-to-end against a real BigQuery instance
(deploy + GRAPH_EXPAND/AGG measure queries, confirming measures are
fan-out-safe). Two cases the offline tests missed:

- Hyphenated GCP project IDs (e.g. `my-proj.ds.tbl`) were emitted unquoted
  and flagged as non-identifiers, producing invalid SQL. Table qualifier
  validation now permits hyphens in a dotted reference, so the whole path is
  backtick-wrapped.
- A MEASURE can only aggregate columns exposed as node properties; a metric
  over a column no field declared produced DDL BigQuery rejects. The emitter
  now auto-exposes any measure-referenced column that isn't already a property
  (new _expr.referenced_columns helper).

Add regression tests for both and document the auto-exposure behavior. The
shipped TPC-DS golden model also deploys cleanly to live BigQuery; golden
output is unchanged.
…ming

The BQ Graph DDL -> Ossie direction was only a not-implemented stub. Since
import is deferred with no concrete use case, drop the stub module, its CLI
`import` subcommand, and the package export rather than ship dead code. The
converter is now cleanly export-only, matching the snowflake/polaris spokes.

Also standardize prose on the approved brand name "BigQuery Graph" (capital G)
across the README, docstrings, comments, and package metadata.
The aggregate feature is just "measures". Update prose to "BigQuery Graph with
measures" across the READMEs, docstrings, comments, and package metadata. The
real doc URL (docs/graph-measures) is left as-is.
The docstrings had overlong summary lines that forced a blank-line split with
dangling continuation fragments, plus several mid-phrase wraps. Rewrite them as
proper PEP-257 docstrings: a short complete summary line, then cleanly wrapped
bodies, all within 80 columns. No behavior change.
The `_common.py`/`_expr.py` split was inherited from the bidirectional
databricks spoke, but here both files were consumed only by the emitter and
`_common.py` had degenerated into a junk drawer (error type, validation, YAML
loading, and Ossie accessors under a vague name), plus two dead symbols
(`is_simple_identifier`, `VENDOR`).

Reorganize into two clearly-named modules:
  - converter.py       (was ossie_to_bq_graph.py) -- the emitter, plus
                       ConversionError and the input loading/validation/accessor
                       helpers that only it uses
  - sql_expressions.py (was _expr.py) -- the literal-aware SQL scanning, the one
                       genuinely reusable, self-contained concern

Delete _common.py and its dead code; rename the test file to match. No
behavior change; 40/40 tests pass.
sql_expressions.py hand-rolled SQL analysis with regex, using two
different string-literal strategies for the same concern and a
hand-maintained keyword/type blocklist. Rewrite the three helpers
(referenced_datasets, strip_qualifier, referenced_columns) on top of
sqlglot's parser, reading facts from the expression's column nodes.
This is the house style already used by the dbt and gsf spokes, and
lets the grammar handle string literals, function names, keywords, and
type names instead of special cases.

Emitted DDL is unchanged (sqlglot preserves the expressions verbatim in
these cases, including the `||` operator).

Also drop the sql_body() test helper: the golden .sql fixture no longer
carries an ASF header, so the export test compares byte-for-byte against
the raw fixture. Fixtures are test data and the repo enforces no license
header on them.
return f'"{escaped}"'


def _describe(description, synonyms):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is off. In BQ Graph, you have OPTIONS. OPTIONS contain synonyms. Please do a thorough job to understand what's possible. Do not put a whole bunch of hacks like this.

return "\n\n".join(parts) if parts else None


def _options_clause(description, synonyms):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Such a short function, let's inline it.


# Apache Ossie spec version this converter targets (see core-spec/).
#
# NOTE: this is an exact-match check. Like the databricks spoke, this converter

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not mention the names of other products other than BigQuery in your folder. For example, Databricks is mentioned here; please do not do this.

datasets = {}
for ds in dataset_list:
name = require_str(ds, "name", f"Model '{model_name}': dataset")
if name in datasets:

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is essentially validating the model. Are there already existing validation modules in Ossie codebase? We need to make sure our converter is reusing the rest of the codebase as much as possible.

def _place_metric(model_name, metric, datasets, measures_by_dataset):
name = require_str(metric, "name", f"Model '{model_name}': metric")
expr = pick_expression(metric.get("expression"), f"metric '{name}'")
if expr is None:

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it have to be BigQuery or ANSI SQL dialect? If it's Databricks, for example, is it possible to have such an input? If so, shall we convert that into BigQuery dialect using SQLGlot?

# specific language governing permissions and limitations
# under the License.

"""Dataset-qualifier analysis for the SQL expressions in a semantic model.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is so short. Shall we consider inlining it to the converter.py?

Comment thread converters/bigquery/tests/_util.py Outdated
# under the License.

"""Shared test helpers: fixture loading."""

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is so short. Can we inline it? What's the point of having a separate file?

…ialect transpile

- Emit description and synonyms as native BigQuery Graph OPTIONS
  (`description="..."`, `synonyms=[...]`) on the element DEFAULT LABEL and on
  properties, instead of folding synonyms into free-text descriptions. Verified
  the syntax (incl. `synonyms` array and `DEFAULT LABEL` placement) against live
  BigQuery.
- Delegate model parsing and structural validation to the core apache-ossie
  pydantic models (OSIDocument.model_validate), reusing the shared schema rather
  than re-implementing require/require_str/version handling.
- Transpile non-BigQuery/ANSI SQL dialects (e.g. Snowflake, Databricks) to
  BigQuery with sqlglot instead of skipping them; only non-SQL dialects
  (MDX/TABLEAU/MAQL) or a missing expression are skipped.
- Inline the small sql_expressions module into converter.py and the test
  _util helper into the test module.
- Drop other-product references from the converter folder.

All 43 tests pass; the regenerated golden DDL is accepted by live BigQuery.
# Ossie SQL dialects mapped to their sqlglot names. BigQuery and ANSI_SQL are
# used verbatim; the rest are transpiled from these names to BigQuery. Non-SQL
# dialects (MDX, TABLEAU, MAQL) are absent -- they have no BigQuery rendering.
_SQLGLOT_DIALECT = {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not future compatible because there will be more dialects added. In that case, we will have a stale list. How do we do it in a more future-proof way?

return _INDENT * depth + text


def _table_group(keyword, entries):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function names are very ambiguous. What do you mean by a table group? In your comment, you said "render clauses and their entries". They don't sound like table group to me. Please make sure all function names are clear.

…learer names

- Resolve transpilable dialects dynamically via sqlglot's registry instead of
  a hardcoded SNOWFLAKE/DATABRICKS table and dialect-preference tuple, so a SQL
  dialect added to the core spec is picked up with no code change and a non-SQL
  dialect is skipped automatically.
- Rename ambiguous layout helpers (_table_group -> _render_tables_clause,
  _properties_block -> _render_properties_clause) and tighten their docstrings.
- Scrub remaining non-BigQuery product names from docstrings and README, and
  drop the stale hardcoded dialect lists there.
return "\n".join(lines)


def _render_field_property(dataset, field):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For each function, let’s have a proper comment to describe what it does.

return "\n".join(lines)


def _render_field_property(dataset, field):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does a field property mean? I have not heard that term before.

return f'"{escaped}"'


def _options_clause(description, synonyms):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function name is inconsistent with other function names we commented before. So let's really make sure to go over all the function names in the converter and make sure they are clearly reflecting what they do, not only for this function, but for all functions in the converter. We really have to write clean code. This is not high enough quality.

return f"DEFAULT LABEL {opts}" if opts else None


def _description_of(obj):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By looking at this function name, I don’t know what it does. Is it a description of what? What is the input type? This is very confusing. You can do much better.

return None


def _referenced_datasets(expression, dataset_names):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function name doesn't really reflect what the comment says of what it does. I find the code hard to read. I care about the code quality. Make sure the function names are really readable and really reflect what it does. Please do a better job in code quality.

return name


def _transpile(sql, read_dialect, what):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter of what? It’s really hard to understand what that parameter means and how it’s used.

libei added 3 commits August 9, 2026 20:25
…ction

- Give every function a docstring stating what it does, its inputs, and its
  return, per review.
- Rename functions so the name reflects the action and is consistent across the
  module: the DDL-emitting helpers are all _render_*
  (_options_clause -> _render_options_clause, _label_options ->
  _render_default_label_clause, _string_literal -> _render_string_literal),
  _convert_model -> _render_property_graph, _render_field_property ->
  _render_property_from_field (drops the unclear "field property" term),
  _validate -> _parse_document, _place_metric -> _register_metric_measure,
  _referenced_datasets/_referenced_columns -> _find_referenced_*,
  _strip_qualifier -> _strip_dataset_qualifier, _parse -> _parse_sql,
  _validate_single_root -> _warn_unless_single_root, _line -> _indented_line,
  _qualify_graph/_qualify_table -> _qualify_graph_name/_qualify_table_name,
  _description_of/_synonyms_of -> _element_description/_element_synonyms.
- Rename the opaque 'what' parameter to 'context_label' (and 'key' to
  'field_name', 'obj' to 'element', 'valid' to 'node_names') and document what
  each means.

Pure rename/documentation pass; output DDL is byte-identical (golden test
unchanged) and all 43 tests pass.
BigQuery edge tables can carry PROPERTIES just like node tables. The core
spec has no field slot on a relationship yet, so edge properties are
declared in a relationship custom_extensions entry whose JSON payload
mirrors the core `fields` shape verbatim -- the form a spec-native
relationships[].fields would take, promotable later with no change to
authored models.

Edge fields validate with the shared OSIField model and render through the
same path as node fields (bare column or <expr> AS <name>, optional inline
OPTIONS). Adds a readable golden fixture pair
(edge_properties_ossie.yaml -> edge_properties_graph.sql) plus unit tests;
full suite 52 passing.
Give every metadata-bearing object in both golden fixtures a description
and a structured ai_context (synonyms): the model, every node, every
property, both edges, each measure, and the edge properties. This showcases
how Ossie's semantic AI context flows onto BigQuery's native graph
OPTIONS(description=..., synonyms=[...]) at every level, edge properties
included. Goldens regenerated; full suite 52 passing.
libei added 8 commits August 9, 2026 21:08
The edge-properties custom extension is a BigQuery-specific convention until
the core spec gains an edge-field slot, so it belongs to the GOOGLE vendor
rather than the vendor-neutral COMMON namespace. Make the tag meaningful:
the converter now reads edge properties only from the GOOGLE-owned extension
and leaves any other vendor's extension untouched. Full suite 53 passing.
The hand-authored edge-properties example is a property graph, so drop the
data-warehouse framing: rename the fixture to orders_ossie/orders_graph and
describe nodes as entities (an order, a customer) rather than fact/dimension
tables. Both graphs target BigQuery, so the fixtures and test helpers now
declare the BIGQUERY expression dialect instead of ANSI_SQL (the
BIGQUERY-over-ANSI preference test keeps its explicit dialects). Output is
unchanged for tpcds; the orders golden is regenerated for the new descriptions.
Validated the generated DDL against live BigQuery. BigQuery treats a graph
label or property name as an implicit synonym of itself and rejects a CREATE
PROPERTY GRAPH whose synonyms list repeats that name, or lists the same synonym
twice -- both compared case-insensitively. The converter now drops such
synonyms (keeping the first spelling) and warns, so the emitted DDL is always
accepted. Also fixes the orders fixture, whose 'orders' synonym collided with
its label. Verified live: both goldens and a collision-laden model now deploy
cleanly, edge properties are queryable, and measures aggregate once per key
under fan-out.
A plain foreign-key relationship is many-to-one: the edge is backed by the
`from` table and each `from` row links to at most one `to` row. A many-to-many
link (a student takes many courses; a course holds many students) needs a
junction table, which BigQuery models as an edge whose SOURCE and DESTINATION
keys both sit on the junction and REFERENCE two different nodes.

Add an additive `association` block, carried in the same GOOGLE
custom_extensions payload that already holds edge properties: it names the
junction table and its two foreign-key column sets, while the relationship's
from_columns/to_columns stay the referenced key columns on the endpoint nodes.
The direct foreign-key path is unchanged.

GRAPH_EXPAND only walks many-to-one / one-to-one edges, so a many-to-many edge
is consumed with MATCH, carries no measures, and is excluded from the
single-root check. Verified against live BigQuery: the generated golden deploys
and traverses in both directions with its edge properties.

Adds the enrollments_graph golden fixture and unit tests.
Replace the relationship-attached `association` block with a first-class
edge: an edge is modeled exactly like a dataset (its own `source`,
`primary_key`, and `fields`) plus two endpoints (`source_key`/
`destination_key`), each `{columns, node, references}` reading left-to-right
as its SOURCE/DESTINATION KEY ... REFERENCES clause. This mirrors BigQuery,
where an EDGE TABLE is a NODE TABLE plus two endpoints, and drops the
confusing overload where a relationship's from_columns/to_columns changed
meaning between the FK and junction cases.

Core-spec `relationships` stay as sugar for a many-to-one FK edge; both
relationships and first-class `edges` normalize to one internal shape and
render through a single path, so the tpcds/orders goldens are byte-identical.
First-class edges ride in a model-level GOOGLE custom_extensions `edges` list,
shaped as a future spec-native `edges:` (a sibling of `datasets:`) would be.

- rename fixtures mn_ossie.yaml/mn_graph.sql -> enrollment_ossie.yaml/
  enrollment_graph.sql; regenerate the golden (warning-free)
- rewrite the M:N tests for the first-class edge shape (endpoints, key/
  references defaults, arity + structure validation, root-check suppression)
- README: rewrite the Many-to-many edges section for first-class edges
- live-validated on BigQuery: CREATE + forward/reverse MATCH + edge property
  + genuine M:N (course held by 3 students; students taking 2 courses each)
Every edge is a `relationship`, whatever its cardinality, so a many-to-many
link now lives in the same `relationships:` block as any many-to-one edge
instead of a separate model-level `edges` list.

The native relationship fields name the two node endpoints (`from`/`to` are
the nodes; `from_columns`/`to_columns` are the columns referenced on them).
The one thing the core spec cannot express yet -- the through-table -- rides
in a GOOGLE-owned `relationship` custom extension on that relationship: its
`source`, `primary_key`, and `source_key`/`destination_key` join columns.
`from` renders as BigQuery's SOURCE and `to` as its DESTINATION. The
through-table appears only as `source`; it is never a dataset, since a dataset
is an entity (a node), not a relationship.

Edge properties (1:M and M:N) unify under `relationship.fields`. Both
cardinalities normalize to one internal shape rendered by one path, so the
tpcds/orders golden outputs stay byte-identical.

A relationship has no `description` field, so its human description is
authored as `ai_context.instructions` and rendered on the edge label;
descriptions and AI context are set on every object in the enrollment
fixture. Live-validated on BigQuery (CREATE + M:N MATCH).
Fixes five robustness gaps found in code review:

- Reject M:N intent without a through-table: a relationship extension
  carrying source_key/destination_key/primary_key but no `source` now
  raises instead of silently emitting a wrong one-to-many FK edge.
- Validate relationship (edge) names are unique and distinct from
  dataset (node) names, mirroring the existing dataset-name check;
  duplicates would emit DDL BigQuery rejects at deploy.
- A non-list `fields` in the extension raises a clean ConversionError
  instead of an uncaught TypeError.
- Merge multiple Google-owned `relationship` extensions loss-free:
  `fields` lists concatenate, and any other key redefined with a
  conflicting value raises rather than being silently overwritten.
- Warn when an edge endpoint REFERENCES columns that are not the
  referenced node's KEY (a deploy-time constraint, like the single-root
  check).

Adds regression tests for each and documents the new requirements and
warning.
Add a per-key legend above the GOOGLE relationship custom_extensions in
the enrollment (many-to-many) and orders (one-to-many with edge
properties) fixtures, explaining source/primary_key/source_key/
destination_key/fields and noting why the comments must sit outside the
data: | block.
synonyms:
- takes
- registered for
# The through-table rides here, in a GOOGLE-vendored `relationship`

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just call it edge table.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8a30779 — switched the header and the extension legend from "through-table"/"junction" to BigQuery's "edge table" term (the association table is declared inside EDGE TABLES).

# Its presence is what marks this edge many-to-many;
# a relationship with no `source` is a plain 1:M FK.
# primary_key -- the junction's own key, used to de-duplicate rows.
# source_key -- junction column(s) that join to the `from` node

@libei libei Aug 10, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly here, edge table.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8a30779 — same rename here; source_key/destination_key/primary_key are now described as the edge table's own columns/key.

# untouched. The converter reads these keys:
# source -- the junction/through-table (campus.public.enrollment).
# Its presence is what marks this edge many-to-many;
# a relationship with no `source` is a plain 1:M FK.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a relationship in this extension doesn't have a source, does it mean it's not an M to N? If it's not M to N, should it just fall back to the oasis original spec without using this extension? Please be very clear.

Do we have a test case covering this case? You have an extension for relationships but do not have a source specified.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified in 8a30779. Yes: no source means it is not many-to-many. The relationship then stays an ordinary one-to-many FK edge (from_columns -> to_columns), and the extension may still legitimately carry only fields (edge properties) — that is exactly the orders fixture. It is not a full fallback that ignores the extension; the extension is still read for fields. Supplying the M:N-only keys (source_key/destination_key/primary_key) without a source is a hard error (ConversionError), not a silent 1:M fallback, so ambiguous intent fails loudly.

Test coverage:

  • test_extension_without_source_is_one_to_many (new in 8a30779) — extension with only fields, no source -> asserts the 1:M edge shape (SOURCE KEY (order_id) REFERENCES orders (order_id)).
  • test_many_to_many_keys_without_source_raise — M:N keys but no source -> raises.
  • The orders golden also covers the no-source, fields-only form end to end.

…fixture

Address review on enrollment_ossie.yaml:
- Use BigQuery's "edge table" term for the many-to-many backing table
  instead of "through-table"/"junction" throughout the header and the
  custom-extension legend.
- State explicitly that a relationship whose GOOGLE extension has no
  `source` is NOT many-to-many: it stays a one-to-many FK edge (the
  extension may then carry only edge `fields`), and that the M:N-only
  keys without a `source` are an error, not a silent fallback.
- Add test_extension_without_source_is_one_to_many asserting the 1:M edge
  shape for a fields-only (no-source) extension.
@libei

libei commented Aug 10, 2026

Copy link
Copy Markdown
Owner Author

Review status

Latest batch (enrollment fixture) — addressed in 8a30779:

  • "call it edge table" (x2): renamed through-table/junction -> edge table in the header and the extension legend.
  • no-source semantics: the legend + a new test now make explicit that no source = not many-to-many (stays a 1:M FK edge; extension may still carry only fields), and that M:N keys without source raise. See replies inline.

Earlier batch (converter code, resolved before this push):

Comment Resolved in
OPTIONS should carry synonyms natively, not folded text 83d2306 — synonyms now emit as synonyms=[...]; folding removed
_describe too short / inline it 83d2306 — removed with the folding logic
Do not mention other products (Databricks) 98ae853/683e1e2 — no product names remain in src/
Reuse existing Ossie validation 83d2306 — parsing/validation delegated to OSIDocument.model_validate (pydantic)
Transpile non-BQ/ANSI dialects via sqlglot 83d2306_pick_expression transpiles any sqlglot-known dialect to BigQuery
_SQLGLOT_DIALECT static list not future-proof 98ae853 — resolved via sqlglot.Dialect.get_or_raise, no static list
sql_expressions.py too short / inline 98c422a — folded into converter.py
tests/_util.py too short / inline 98c422a — folded into conftest.py
Ambiguous function names (_table_group, _options_clause, _description_of, _referenced_datasets, etc.) 98ae853/683e1e2 — renamed (_render_tables_clause, _render_options_clause, _element_description, _find_referenced_datasets, …)
_transpile(..., what) unclear param; "field property" term 683e1e2what -> context_label; every function docstringed

All 88 tests pass.

Reorganize the README so a first-time reader can follow one train of
thought: why -> install -> see it work end to end -> understand the
mapping -> run it in BigQuery -> reference -> development.

- Quick start now walks the full author -> convert -> deploy -> query
  loop as numbered steps, instead of dropping the reader into CLI/API
  reference right after their first conversion.
- Group pure look-up material (CLI, Python API, requirements,
  limitations, warnings) under one Reference section at the end.
- Fold the separate "Metrics and measures" section into the
  "Metric -> measure" mapping rule so all mapping rules live together.
- Merge "Deploying" and "Querying" into one "Run it in BigQuery".
- Align the many-to-many prose with the fixture's "edge table"
  terminology and state explicitly that no source means not many-to-many.
No SQL/YAML examples or technical content changed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant