Repository navigation
feat(bigquery)!: the catalogue from INFORMATION_SCHEMA, and one introspect question per family - #5
Merged
Conversation
The shape check now lives where the flag has to be honoured rather than where it is declared, so a value that would silently render the wrong DDL is named instead. The attribute is read directly: whether the field exists is settled by the class, so probing for it with a default would only turn a wrong class into a quiet signed column. CI installs the core library from fix/column-constraint-gates so this backend runs against the branch it depends on.
…with getattr The formatter parameters are annotated with the classes that declare these fields, so their presence is statically determined and a probe with a default would only turn a wrong-class call into a quiet wrong answer. Both the materialized-view drop/alter formatters and the CHAR/VARCHAR/DECIMAL type formatters now read the attribute outright.
Mechanical rename wave against core's W1/W2/W3a/W3b/S1-c breaks. The constraint surfaces still reference the enums core's S3 removed, so this commit is an expected red window -- the constraint mixin and its switch test follow in the next commit. W2, statement-level classes -> *Statement (core 34e1fb45): QueryStatement, CreateTableStatement, DropTableStatement, CreateTableAsStatement, CreateTableLikeStatement, CreateTableCloneStatement, CreateTableFromTemplateStatement, CreateSchemaStatement, DropSchemaStatement, CreateMaterializedViewStatement, DropMaterializedViewStatement, RefreshMaterializedViewStatement, TruncateStatement, CreateViewStatement, DropViewStatement, InsertStatement, UpdateStatement, DeleteStatement, MergeStatement, BeginTransactionStatement, and the ALTER/CREATE/DROP TYPE/DOMAIN/DATABASE/SEQUENCE/INDEX/FUNCTION/TRIGGER family, plus CommentOnStatement and the property-graph three. W2 formatter rename (functional, 1 hit): format_begin_transaction -> format_begin_transaction_statement in mixins/transaction.py, where the BigQuery-only WAIT / NO WAIT refusal lives; without it the override would have been dropped and the pair silently rendered. W3a, predicate/clause families (core 890aaaa3): ILIKEExpression -> ILikePredicate, XMLExistsExpression -> XMLExistsPredicate, FulltextMatchExpression -> FulltextMatch. The two dispatch-failure fragments the roundtrip matrix pins were the old formatter spellings (format_xmlexists_expression, format_ilike_expression) and are updated to format_xmlexists_predicate / format_ilike_predicate, because a class with no formatter is reported by the method it could not find. W3b, value/function/wrapping families (core 5dc64fd5): SQLValueExpression -> SQLValue, QualifiedIdentifierExpression -> QualifiedIdentifier, UUIDCastExpression -> UUIDCastFunction, CastExpression -> CastFunction, CaseExpression -> CaseFunction, TrimExpression -> TrimFunction (the BigQueryTrimMixin parameter annotation; the formatter name format_trim_expression is unchanged in core), CollateExpression -> CollateFunction, JSONDocumentExpression/JSONTextExpression -> JSONDocumentValue/JSONTextValue, ArrayValueExpression -> ArrayValue, ArrayExpression -> ArrayOperation, BooleanLogicExpression -> LogicalValue, _TemporalValueExpression -> _TemporalValue. W1, CTE family (core 683a183e): CTEExpression -> CTEClause in the master probe guard. The mixins/capabilities.py docstring reference to format_cte_expression becomes format_cte_clause. S1-c, PASSING: no change needed -- this backend constructs no XML passing clause. WildcardExpression and the format_wildcard override are deliberately NOT touched here; the wildcard split is its own wave. Verified: import smoke over every module in the backend package, 33 ok / 0 failing; compileall src+tests exit 0; ruff on the changed file set is identical to the HEAD baseline (the F401 line in test_expression_fields_match_formatters.py changes only the renamed symbol); targeted runs exit 0: test_bigquery_master_probe_guard.py dialect/test_bigquery_materialized_view.py test_bigquery_clause_pair_guard.py ddl/test_create_table_family.py
Core's S3 wave replaced the fat constraint nodes with a production plus a
branch: ``ColumnConstraint(branch=...)`` / ``TableConstraint(branch=...)`` name
the branch, the branch names the kind, and the ``*ConstraintType`` enums are
gone. This commit migrates the construction sites this backend owns and folds
in the constraint capability mixin the B-series draft left uncommitted -- its
grammar evidence, its local declarations and its (self) probes.
Construction sites migrated (src + tests):
test_ddl_source_mixin.py UseConstraint(ColumnConstraintType.DEFAULT, ...)
-> UseConstraint(ColumnDefaultConstraint, value=...)
the same for UNIQUE / CHECK; the table
constraints become TableUniqueConstraint /
TableCheckConstraint branches; the kind
assertions read ``type(item.branch)`` instead
of the removed ``constraint_type`` tag.
test_bigquery_capability_assertions.py
GeneratedColumnExpression (deleted from core)
-> ColumnGeneratedConstraint, which is the same
production with one node.
test_expression_roundtrip_all.py
the ColumnConstraint / TableConstraint /
AddTableConstraint / AlterConstraint
constructors move to ``branch=`` and
``constraint_kind=``.
No construction site in ``src`` needed migrating: this backend builds no
constraint node itself (S3 construction points in src: 0).
The mixin. ``mixins/constraint.py`` is the constraint capability declaration
the draft wrote before S3 and never committed. It is kept whole -- the grammar
transcription, the 文档（待云验） evidence level, the one-notch finding and the
switch values are all preserved verbatim -- and rewritten onto the new API:
* both probes converge to ``(self)``. A probe answers whether the clause
family exists on this dialect; the kind is not part of the question, so the
``constraint_type`` parameter (and the docstring passage that had it routed
through core's deleted ``_call_constraint_capability``) are gone.
* the per-kind answers move to the render path, where the branch class is the
only thing that can name a kind. Two ``_`` helpers carry them --
``_require_bigquery_table_kind`` (CHECK / UNIQUE have no table production)
and ``_require_bigquery_enforcement`` (the marker rides PK / FK; ``ENFORCED``
does not exist here at all) -- and they are called from two new *public*
overrides, ``format_column_constraint`` and ``format_table_constraint``. No
private formatter is added.
* the gap list is rewritten against what S3 actually did: the primary-key
marker is now reachable (the old ``ValueError`` that made it impossible is
gone), so what remains is that the marker is still opt-in (a core printing
decision) and the explicit ``NULL`` modifier is unguarded.
Wiring. ``BigQueryDialect`` gains ``BigQueryConstraintMixin`` twice (import and
MRO) ahead of core's ``ConstraintMixin`` / ``DDLColumnMixin``, so a moved core
default cannot answer instead of the local declaration; several values differ
(``CHECK``, ``UNIQUE``, ``MATCH``, ``DEFERRABLE``, ``ON DELETE``, ``ON UPDATE``
are False here and True there).
The ``OPTIONS(description=)`` column-comment path is untouched: it is
``supports_column_comment``, a separate capability, and it already answers
True.
Behaviour changes, all reached through the public formatters:
* a column `CHECK` / `UNIQUE` was refused already (the local False pins); a
*table-level* CHECK / UNIQUE is now refused by name too, where core gates no
per-kind table switch and would previously have rendered it;
* ``enforced=True`` (any kind, either level) is refused by name, because
BigQuery has no bare ``ENFORCED`` token;
* a BigQuery primary key can now carry its compulsory ``NOT ENFORCED`` (it
could not before: core's per-kind ``ValueError`` turned the request away).
Verification:
* tests/.../feature/backend/ddl/test_ddl_source_mixin.py, exit 0
* tests/.../feature/backend/test_bigquery_capability_assertions.py, exit 0
* tests/.../feature/backend/ddl/test_type_domain_negative.py, exit 0
* testsuite feature/ddl shared contract (run from the core worktree), exit 0
* import smoke over every module in the backend package: 33 ok / 0 failing
The switch-contract tests (value table, evidence markers, declaration and
``(self)`` guards, the render-path narrowing assertions) follow in the next
commit; per the wave guide the constraint mixin's code and its probes are one
unit, so only the contract test is held back.
… protocol The contract test for the constraint capability declaration committed with the S3 migration. Nothing here changes behaviour; the code and its probes were one unit in the previous commit (per the wave guide), so this one adds the guard that makes the declaration load-bearing rather than decorative. Evidence level: 文档（待云验） -- official grammar only, not server-verified, stated in the file header and asserted per switch. There is no BigQuery instance reachable from this environment and the CI emulator (goccy/bigquery-emulator on localhost:9050) implements neither of the two ALTER TABLE constraint statements, so the emulator cannot adjudicate these switches either. What it pins: * the value table -- every switch this backend relies on, grouped the way the grammar groups them (column_schema / constraint_definition / ALTER TABLE); * declared-by-this-backend -- the first class in the dialect's MRO that has the name in __dict__ is inside this package. Eight of these answers used to be inherited core defaults that are the exact opposite of BigQuery's grammar (CHECK, UNIQUE, MATCH, DEFERRABLE, both referential actions, and the DDLColumnMixin ON DELETE / ON UPDATE pair); * probes take no argument -- (self) and nothing else, checked on the bound method so a defaulted parameter cannot slip back in behind ``= None``; * the 待云验 marker in every switch docstring, and the module's plain-English restatement of the evidence level; * the render-path narrowing -- a PK/FK takes the compulsory NOT ENFORCED, the bare ENFORCED token is refused on all four kinds and both levels, and a table-level CHECK / UNIQUE is refused by name (core gates no per-kind table switch, so the refusal is this dialect's own formatter); * the remaining gaps are recorded in the mixin's module docstring, not only in the test. The gate ordering is asserted too: the kind refusal wins over the ``CONSTRAINT <name>`` prefix, so a named CHECK is refused for its kind rather than for its name. Verification: tests/.../feature/backend/test_bigquery_constraint_switches.py, exit 0 (166 passed); ruff on the file, clean.
Core split the single asterisk into two by where it sits in the grammar (spec 4.10): ``AllColumnsWildcard`` for the select list and RETURNING, ``RowCountWildcard`` for ``COUNT``'s argument. The two answer different questions -- ``COUNT(*)`` counts rows, ``t.*`` expands columns -- and the old ``WildcardExpression`` / ``Wildcard`` node no longer exists. Backend surface: * ``mixins/dql.py`` carried a ``format_wildcard`` override that re-spelled ``table.*`` by reading ``expr.table`` / ``expr.table_need_quote`` -- a shape the split removed. The override is **deleted rather than renamed**: the qualifier now rides a ``NamedRelationRef`` on the node, and core's ``format_all_columns_wildcard`` renders the name the query itself uses (``rel.*`` or the alias), which is the same output the override produced. There is nothing BigQuery-specific left to say, so a renamed override would have been a copy of the default. (Verified: bare ``*``, ``rel.*`` and ``alias.*`` all render through core.) No ``COUNT``-position call site exists here, and BigQuery's ``* EXCEPT`` / wildcard-table spellings are backend-specific by the guide and are not nodes core provides, so no ``format_row_count_wildcard`` override is added. Docs: ``schema_namespace.md`` (en/zh) described the deleted ``WildcardExpression`` and its ``format_wildcard`` discard of ``schema_name``. The passage is rewritten to the two nodes and the relation-carried qualifier, and the one place a wildcard genuinely cannot carry a dataset is recorded as the backend-specific wildcard table (``dataset.*``), which stays backend-side. Verification: * every backend test except the roundtrip matrix, exit 0 (the matrix's wildcard rows are the next commit's business, together with the other new core classes). * bare ``AllColumnsWildcard``, ``AllColumnsWildcard(relation=...)`` and ``FunctionCall(COUNT, RowCountWildcard)`` render ``*`` / ``o.*`` / ``COUNT(*)``. * ruff on the changed files: clean.
The matrix is the exhaustiveness check the round's CI runs first, and three core waves left it short: S3's two constraint branch bases, S1-c's ``PassingClause``, and the LIKE/ILIKE constructors core's own dummy2 matrix already carries. It is now all green, run in full rather than sampled. Rows added: * ``ColumnClause`` / ``ColumnConstraintBranch`` / ``TableConstraintBranch`` -- the bases S3 introduced. Each names no kind and so declares no ``format_method``; classified as ``LEGITIMATE_NOT_IMPLEMENTED`` with the ``_NO_DISPATCH`` message, which is what the base's own ``NotImplementedError`` says, not a capability the dialect lacks. * ``xml.PassingClause`` -- S1-c's clause node. BigQuery has no SQL/XML, so it belongs in ``LEGITIMATE_NON_RENDERS`` and its ``format_passing_clause`` in ``UNMODELLED_FORMATTERS`` under the existing SQL/XML group, which is what the omission reports. * ``predicates.LikePredicate`` / ``predicates.ILikePredicate`` -- registered constructors, copied from core's dummy2 matrix so the two agree. The generic guess fills ``op`` (LIKE's operator string) and the operands with bare strings, so the formatter calls ``.to_sql()`` on a ``str`` and raises ``AttributeError``. The constructors are shape-for-shape what dummy2 uses. Two dispatch-failure fragments were also the old formatter spellings, fixed in the rename wave: ``format_xmlexists_predicate`` (was ``format_xmlexists_expression``) and ``format_ilike_predicate`` (was ``format_ilike_expression``), in both the per-class pins and the ``UNMODELLED_FORMATTERS`` rows. Verification: tests/.../feature/backend/test_expression_roundtrip_all.py, full run, exit 0 (the four UUID/CustomType skips are the pinned UNCONSTRUCTIBLE entries). Coverage report: 313 classes -- 189 rendered, 54 unsupported, 51 no-formatter, 15 not-implemented, 4 unconstructible. The whole feature/backend tree, exit 0.
The constraint-gates work this branch carries is not on any release branch, so neither dependency can come from one. CORE_PKG already points at fix/column-constraint-gates; TESTSUITE_PKG still pinned release/v1.0.0.dev17, which does not carry the W2/W3/S3 adaptation the shared feature/ddl contract now requires. It moves to the same branch, matching the core's own CI (python-activerecord@fix/column-constraint-gates), so the two refs move together. The env comment above is updated to say so plainly, rather than continuing to describe both refs as release branches. Verified locally against the pinned refs' working trees in the central environment: testsuite feature/ddl shared contract, exit 0.
Carry the W3c class renames, the R1 signature alignment and the R2a arithmetic/concatenation renaming through the backend sources and tests. The roundtrip matrix moves off the retired class-based registry onto the serialization class index, and registers the two core classes whose op is a closed word set.
Wave P of the 2026-10-17 value/literal/reflection guide. 15 probing sites
cleared, rg '(getattr|hasattr)\(\s*(expr|node|clause|item)\b' src now has
zero matches.
Migration surface (old -> new):
- mixins/materialized_view.py, 11 sites (the guide's whole 83-255 span):
* or_replace / if_not_exists / partition_by / cluster_by / options --
declared only on BigQueryCreateMaterializedViewExpression, so the
create formatter now dispatches once by isinstance on that class and
then reads each field directly; a generic CreateMaterializedViewStatement
declares none of them, so its grammar simply has no such clause.
* column_aliases / tablespace / storage_options / with_data / no_data /
query -- declared on the core CreateMaterializedViewStatement, now
read directly in _reject_unsupported_clauses and
_materialized_view_query_sql (both annotated with that class, matching
the postgres fcc9939 form-A rationale).
- mixins/constraint.py, 4 sites beyond the guide's gate (the probing
variable is named `constraint`, which the gate pattern does not cover,
but the fields are expression-contract declarations all the same):
* enforced / not_enforced / branch on _require_bigquery_enforcement,
now read directly off the ColumnConstraint/TableConstraint production
nodes that declare them; parameter annotated with the union.
* branch on _require_bigquery_table_kind, read directly; parameter
annotated TableConstraint (its only caller).
Overridden formatters: none renamed; the two constraint refusals and the
create/drop/alter/replica materialized-view formatters keep their names.
Behaviour change (tightening): a non-constraint object reaching
_require_bigquery_* now fails loudly with AttributeError instead of being
silently treated as a constraint without traits; a non-statement reaching
the materialized-view helpers likewise. No rendered SQL changed -- the
default branches that fired were all the declared defaults.
Evidence: PYTHONPATH=src;tests pytest
feature/backend/dialect/test_bigquery_materialized_view.py (52 passed),
feature/backend/test_bigquery_constraint_switches.py (229 passed),
feature/backend/test_bigquery_ddl_improvements.py +
test_bigquery_clause_pair_guard.py (46 passed), all exit 0;
ruff check on both files clean. Core import self-check OK before each run.
…llings Wave V of the value-literal reflection guide: wire the core TypedValueMixin and this backend's spellings onto the dialect (guide 4.1). No BigQuery instance is reachable from this repository, so every form is a rendering assertion transcribed from the official reference pages (lexical structure and syntax / data types, read 2026-10-17, not server-verified). Wiring: BigQueryTypedValueMixin ahead of core TypedValueMixin, both ahead of SQLDialectBase; mixins/__init__ exports it. The in-flight draft had the class bases without the import in dialect.py -- fixed (collection NameError). Overrides, all DDL-position only (data positions keep placeholder + host value): binary = b'\\x01\\x02' bytes literal, every byte a \\xhh escape (valid for any bytes); UUID = bare quoted text (no UUID type, GENERATE_UUID() returns STRING); INTERVAL = the unquoted single-part form INTERVAL n UNIT, INT64 step mapped over the timedelta's days/seconds/microseconds and spelled in the largest dividing unit down to MICROSECOND (YEAR/QUARTER/MONTH are calendar units a timedelta cannot state; WEEK is exactly seven DAYs); Decimal = NUMERIC '...' / BIGNUMERIC '...' keyword form (bare numeric text is a FLOAT64 literal) with NUMERIC = 29 integer digits at scale 9 and BIGNUMERIC = the documented -5.789604...9968E+38..5.789604...9967E+38 bounds (the 39th integer digit is the partial one, the negative bound one unit past the positive); date/time = keyword-prefixed typed literals, naive datetime spelled DATETIME '...' (TIMESTAMP would silently pin UTC), aware datetime TIMESTAMP '...' with its offset; an aware time is refused rather than silently zone-stripped. Dialect-wide escape policy: _escape_sql_string overridden to GoogleSQL's backslash escaping (the \\xhh/\\' table; quote doubling appears nowhere in it and any sequence not in it is an error), so every inline string this dialect renders -- value literals, OPTIONS entries, comments -- is backslash-escaped. dql.py's DATE_DIFF part site now routes through self._escape_sql_string instead of pinning SQLDialectBase's quote doubling. Behavior changes (DDL-position): inline strings containing quote or backslash render backslash-escaped instead of quote-doubled (dialect-wide); date/time/datetime/timestamp inline literals carry the type keyword; Decimal inline literals are the NUMERIC/BIGNUMERIC keyword form; aware times, Decimals past scale 38 or past the documented magnitude bounds, and an INTERVAL step outside signed INT64 now fail explicitly with ValueError. Drive-by pre-existing repair, found by ruff F821 on a changed file: format_column's refusal in dql.py has raised NameError since 1cf7c46 because UnsupportedFeatureError had no import in scope. The module-level import added here (the same pattern ddl.py and transaction.py already use) fixes it; the two red tests it caused (test_column_schema_validation.py::TestBareColumnWithSchema, test_expression_fields_match_formatters.py bare-column case) go green. Tests (exit codes): new test_bigquery_typed_values.py 56 passed, exit 0; feature/backend -k 'typed or literal' 59 passed, exit 0; feature/backend/dialect/ whole dir 253 passed, exit 0; ruff on all changed files clean. The roundtrip matrix is not collected in the -k run: core serialization has moved auto_register_builtins (now default_registry) since the guide was written and the matrix still imports the old name; the wave-4 reconciliation repairs it against the then-current core HEAD.
…ction waves The classed-function wave (M1) audited first, and found nothing to change in this backend, so it rides here as a no-op: BigQuery's LENGTH counts characters, which is what the core's format_char_length_function spells by default, so no char_length override belongs here; this backend overrides no format_char_length_function/format_count_function, defines no format_function_call and no INSERT VALUES source of its own (rg over src: zero stale classed-entry overrides, zero FunctionCall call sites, zero format_insert_statement/format_binary_arithmetic_expression overrides), so there was no code to migrate. The matrix itself needed three repairs against the current core HEAD (e57c6702). Core's serialization moved auto_register_builtins into the lazily built default_registry and dropped registered_expression_classes, which broke this file at import: the deserialization-coverage test now resolves each class's kind through default_registry().resolve_node and the ten eager warm-up calls are gone (register_all warms the registry itself). The testsuite dropped its round-trip harness in e05edb0 with the direction that round-trip coverage moves to the owning repository, so the harness is vendored here as expression_roundtrip_harness.py (from the same testsuite@4ef2551 source mysql vendored, adapted to the split registry the way mysql's copy is, plus one deliberate drift fix: zip() gains strict=False so B905 stops firing while the length assert above stays the check that reports). The third repair is the reconciliation itself: - Constructors: the seven value classes whose host type an int cannot state (BinaryValue/BooleanValue/DateValue/StringValue/TimeValue/TimestampValue/UUIDValue) get real host values; the six new scalar string nodes (Chr/Left/Right/Space/Substring/Overlay) get real operands their SQLValue forward refs defeat the guesser on; ExtractFieldExpression and _ExtractBase get a real field and source. Two stale factories follow the current signatures (ArrayType's element_type is keyword-only now; ValidateConstraint names its constraint positionally) and the dead graph.EdgeTable registration goes -- core renamed that element to EdgeTableDefinition, which the heuristic constructor builds. - Verdict tables: UNCONSTRUCTIBLE loses CustomType (its raw parameter is required now, so it builds and refuses through the generic-type probe like the other unsupported types) and keeps the UUID trio, which still refuse while constructing. LEGITIMATE_NON_RENDERS moves graph.EdgeTable to graph.EdgeTableDefinition with the format_edge_table_definition fragment and pins declared._Annotation as (AttributeError, has no attribute 'alias'): the shared base of the declared-result annotations dispatches to the one formatter those annotations share, which reads the alias only a concrete annotation carries. UNMODELLED_FORMATTERS renames format_edge_table to format_edge_table_definition and adds one group for the empty 'format_' the TypedValue category base dispatches to -- not a feature BigQuery lacks, but the row keeps the unmodelled list exact in both directions. ExtractFieldExpression and _ExtractBase render through the shared format_extract_expression, so they take no table rows. Coverage after reconciliation, 390 classes (core + this backend): rendered 265, no-formatter 52, unsupported 54, not-implemented 15, unmodelled 1, unconstructible 3. The surfaces of values.py are abstract and are not collected, as the wave guide requires. Tests (exit codes): the full matrix file 396 passed / 3 designed skips (the UNCONSTRUCTIBLE UUID trio, each named with its reason), exit 0; TestMatrixIntegrity alone 9 passed, exit 0; ruff on both files clean. Also fixed in passing: BaseExpression was imported but unused in this file (dead already at HEAD; F401).
Core replaced TypedValueMixin and the twelve *Value classes with one Literal node whose value kind is a LiteralKind member, and moved every tier-2 spelling into format_literal. This backend's import of TypedValueMixin no longer resolved, so the dialect would not load. mixins/typed_value.py becomes mixins/literal.py, holding BigQueryLiteralValueMixin with a single format_literal override composed ahead of SQLDialectBase. The spellings it used to dispatch to per-kind formatters keep their exact text, now reached by host type: bytes -> b'\\xhh', UUID -> bare quoted text, timedelta -> INTERVAL n UNIT, Decimal -> NUMERIC / BIGNUMERIC, date -> DATE '...', time -> TIME '...', datetime -> DATETIME / TIMESTAMP by whether a zone is held. Every other host type falls through to the base, which is where NULL / booleans / numbers / strings and the literal-construction protocol live. test_bigquery_typed_values.py becomes test_bigquery_literal_values.py on the same assertions, re-expressed as the two positions one node answers (bind by default, inline=True for DDL) instead of the retired class-level inline_params strategy. The documented, not server-verified evidence recorded in the mixin docstring is carried over unchanged. Verified: dialect imports; test_bigquery_literal_values.py 61 passed; ruff clean.
Kind census run with the core script from a temporary copy carrying this
backend's expression package and dialect class in BACKENDS (registering it in
the core script is a core-repo change, reported rather than made here):
core kinds 357, bigquery override 0, bigquery custom 4.
Override zero is the rule that matters most: a backend may replace the
rendering of a kind core ships, never the node class behind it, because two
classes under one kind leave the deserializer no way to choose. This backend
adds four materialized-view nodes and overrides formatters; it replaces
nothing.
The census's finding: all four custom kinds spelled the backend's name as two
words -- big_query_create_materialized_view_expression and the other three.
BigQuery is one word, and the general derivation (a run of capitals counts as
one) splits at the camel-case seam instead. The fix is the data fix sqlite,
mysql, clickhouse and sqlserver already made: the expression package declares
KIND_RESERVED = ("BigQuery",) for every node it defines, so the kinds read
bigquery_create_materialized_view_expression and the four are parallel to
sqlite_integer_type / mysql_big_int_type. KIND_RESERVED is data, not behaviour
-- no class overrides kind(), so class name, kind and formatting entry keep one
source. The loop walks the package's own modules rather than its __all__, so a
node defined but not re-exported cannot keep the split spelling.
Section 11.1 order audit, run mechanically over BigQueryDialect's 108-entry
MRO: of the 46 format_* methods a BigQuery mixin defines, none is reached after
an earlier definer of the same name, so no override loses the lookup. That
includes the one this round added -- BigQueryLiteralValueMixin ahead of
SQLDialectBase, because it overrides format_literal. Recorded because it is the
check that has to keep holding.
Verified: the census re-run reports 0 overrides / 4 customs, every one
bigquery_*; tests/.../feature/backend apart from the two modules core's own
moves left red exits 0 -- test_expression_roundtrip_all.py still names the
retired *Value classes and ddl/test_type_domain_negative.py still imports the
retired DdlLiteral, both reconciled in the matrix commit that follows; ruff
clean.
…errides Three checks, one per clause of expression-dialect 11. **One entry per kind, named after its kind (§2, §3).** The four materialized-view nodes were named *Expression while their formatters said _statement, and all four dispatched to *core's* entry names -- format_create_materialized_view_statement and its sibling -- so one entry took two different contracts and told them apart by isinstance. Two of those contracts differ in fields the other does not declare at all (OR REPLACE, IF NOT EXISTS, PARTITION BY, CLUSTER BY, OPTIONS, AS REPLICA OF), which is the shape §2 names: an entry branching on the object it was handed to decide *which* contract to render. The four classes are now *Statement, each kind derives its own entry (bigquery_create_materialized_view_statement & co), and the two entries over each shared shape meet in one private helper -- _create_materialized_view_sql takes the five BigQuery-only clauses as keyword arguments, so nothing reads an attribute off a node that may not carry it. The protocol follows: it declares the four backend entries beside core's two, which remain BigQuery's answers because this mixin outranks core's ViewMixin. Rendered SQL is unchanged, checked rather than argued: the seven statements from the old and the new shape are byte-identical, including OPTIONS(...), PARTITION BY, project.dataset qualification and AS REPLICA OF. **Enum spelling (§11.3, §12.1).** BigQueryMaterializedViewOptionValueType was a plain Enum, so str() gave "BigQueryMaterializedViewOptionValueType.BOOLEAN" -- the one thing a plain-JSON artifact cannot carry. It is (str, Enum) with __str__ = str.__str__ now, the spelling sqlserver, mysql and clickhouse use. BigQueryMaterializedViewOption had the same gap and gets the same line. **Override order (§11.1).** The composed dialect's MRO, audited mechanically in the previous commit, already had the arrangement the clause requires: 46 format_* entries defined by a BigQuery mixin, none reached after an earlier definer of the same name. This commit's new entries keep that, since every one of them is declared on BigQueryMaterializedViewMixin, which precedes ViewMixin. New: dialect/test_bigquery_kind_identity.py pins the three sources for every node this package defines -- entry == "format_" + kind, kind starts with the backend name, class name ends with the statement nature -- and pins which mixin owns each of the six entries. It would have failed on all four nodes before this commit. Verified: dialect/schema/protocol/clause-pair-guard suites green (441 passed); the new identity file 19 passed; ruff's finding set is identical to HEAD (19 findings, all pre-existing in these directories).
The org rules (expression-dialect §12) say it in three sentences: expression modules are independent of I/O, so their tests start no connection, create no database and touch no file; every case carries its own dialect instance and shares no mutable state; and therefore tests run in parallel. The first of those was already true here, and worth recording rather than assuming: the three pure-render locations -- feature/backend/expression, feature/backend/dialect and the round-trip matrix -- contain no client construction, no execute() and no scenario fixture. This backend's literal spellings were never server-verified (no BigQuery instance is reachable), so wave ② wrote the CREATE TABLE / INSERT / readback cases as *rendering* assertions from the start; there is nothing to migrate out. The probe names that look like a connection -- api_endpoint, _ensure_client -- are in backend/, the emulator scenario loader, and the emulator row-count fallback tests, which is where they belong. The second half of §12 is configuration, and it was missing. Parallel is now the default rather than something each run remembers to ask for (§12.1). The worker count is read by pytest-xdist before any conftest loads, so a conftest cannot inject -n; the default is a repository-local plugin, tests/plugins/expression_xdist.py, registered from pyproject.toml and therefore active for every run here. It is narrow on purpose: it enables nothing when pytest-xdist is absent, it acts only when every target is a pure-render one (a whole-repository run, a mixed run, a backend-directory run and the emulator scenario loader keep the serial default), it never overrides an explicit -n, it never overrides an explicit --dist, and it refuses --dist=loadscope and --dist=loadfile outright because those bind whole modules together and hide the case-to-case dependencies parallelism exists to expose. -n without --dist gets --dist=loadgroup, the one scheduling the rules allow. The covered locations are three, not one, because this repository's pure-render tests are spread by necessity: dialect/ holds the same kind of test as expression/, and test_expression_roundtrip_all.py walks core's package *and* this backend's, so it cannot live inside either subject directory. feature/backend/expression/conftest.py records the three properties, says where the parallel default is wired and why it cannot be wired here, and lists the three covered locations so the next person to add a case knows which ones it has to keep. No case carries xdist_group: there is no I/O in the area to serialise around, and the mark is reserved for a case that genuinely cannot share a worker, with the reason beside it. Evidence (central venv, PYTHONPATH=src;tests): - feature/backend/dialect (single file, and the whole directory): exit 0, and the run reports "bringing up nodes" -- the plugin's own default firing, with no -n on the command line. - feature/backend/expression: exit 0, same default. - expression + protocol together (a mixed run): exit 0, no worker pool, which is the scope rule keeping every other directory serial. - --dist=loadscope against dialect/: refused by the plugin with the §12.1 reason, exit 1. - ruff on tests/plugins and the expression area: clean. Not run here: the round-trip matrix, which is red before this commit for a reason that belongs to the next one -- it still imports core's retired *Value classes. It is the matrix commit's subject.
…t tree
Reconciled against core 4774c502 (f2bdf386 before it). The matrix was red at
HEAD for four independent reasons, all of them core-side moves this backend had
not followed, plus one that only surfaced once those were cleared.
The value family is one node now. Core replaced TypedValueMixin and the twelve
*Value classes with one Literal whose kind is a LiteralKind member, so the
module imported seven names that no longer exist. The matrix registers one
representative Literal instead: the single node carries every kind, and the
kinds an int cannot state are the literal-spelling file's subject, not this
file's. The unmodelled-formatter row for the retired TypedValue base goes with
them -- the dispatch no longer names an empty 'format_' for anything.
ddl/test_type_domain_negative.py imported DdlLiteral, which the same core move
retired: a DDL position now says inline=True on the Literal itself, and the
CHECK condition is written that way. ArrayType's element_type is keyword-only
now, so that construction names it.
The five ALTER TABLE action classes lost their bare aliases (AddColumn ->
AddColumnAction & co), and the harness's AlterTableStatement constructor and
graph helpers still named the retired spellings -- EdgeTable in particular,
which core renamed EdgeTableDefinition, so that helper is gone rather than
renamed (a helper matching nothing is worse than no helper). The matrix's
constructors follow the same renames.
Then the classes whose introspective guess the core's constructors began
refusing:
- ComparisonPredicate normalises its operator (core 9659c483), and the guess
hands the slot a string that is not one; registered.
- TableCheckConstraint and ColumnGeneratedConstraint are DDL positions that
refuse a bind parameter, so the guess's bind-bearing comparison cannot render
them -- two columns compose a predicate with nothing that binds. The generated
constraint also needs STORED, because the node defaults to VIRTUAL and this
dialect's probe refuses VIRTUAL.
- The ten declared-result annotations read the alias of their source, and a
Literal is deliberately not aliasable; a Column is what the node is for.
- DataType's pin followed the dispatch's new generic spelling ("does not
support the generic type None") rather than the private one it used before.
Finally, four claims the matrix could not express before. A class can render and
still lose one encoding, which "does not render" would misreport and a failure
would blame on BigQuery. LEGITIMATE_NON_ROUNDTRIPS records each such class per
channel, with the type and message fragment, and the new
test_pinned_non_roundtrip_refuses_as_pinned asserts it in both directions -- a
pinned channel that starts working fails the run, so a core fix cannot leave a
stale entry. The twelve entries are core's to fix: DateTimeField/IntervalUnit
inherit (str, Enum) without str.__str__, so the XML channel writes
"DateTimeField.YEAR"; CreateTableCloneMode, ViewOptions, ReferentialAction and
GeneratedColumnType are host objects the plain-JSON artifact cannot name.
Bind parameters are compared as sequences rather than as the same container:
the artifact carries a tuple as an array, and Subquery's formatter passes its
query_params through as whatever it received, so a restored render hands back a
list where the original handed a tuple. The values are the contract; the
container is not.
Coverage at 4774c502: 376 classes collected (core + this backend), 253 rendered,
53 unsupported, 52 pinned non-render, 15 not-implemented, 3 pinned
unconstructible, 12 pinned non-roundtrip per channel.
Evidence (central venv, PYTHONPATH=src;tests):
- test_expression_roundtrip_all.py, full matrix under the parallel default:
373 passed / 3 skipped, exit 0 (the three skips are the UUID nodes core
refuses at construction, pinned in UNCONSTRUCTIBLE).
- feature/backend, the whole directory: 1333 passed / 3 skipped, exit 0.
- ruff on the three changed test files: clean.
The org rules (expression-dialect §16.2) make SQL text the formatter's business: a backend may collect parameters into a node and execute what that node renders, but assembling the statement itself is a call site doing a formatter's job. This backend had four such sites, two in each of the sync and async backends -- the PK probe and the emulator's row-count probe -- each an f-string that pasted a rendered relation into a hand-written SELECT. The consequence was not stylistic. The relation's quoting was restated by hand inside those statements, in code that never saw the dialect's escaping, so a dataset or table name containing a backtick produced SQL the server refuses. The same two statements also disagreed with themselves about their own projection: `AS next_id` was unquoted while the relation beside it was quoted. backend/qualified_names.py already owned naming a relation; it owns the two statements now, as nodes: - _next_pk_expression builds SELECT COALESCE(MAX(pk), 0) + 1 AS next_id over the relation, from MaxFunction, CoalesceFunction, ArithmeticOperation and QueryStatement. The zero and the one are Literal values marked inline=True: they are constants of the statement, not caller data, and inlining keeps the text identical to what the f-string produced. Nothing binds. - _count_expression builds SELECT COUNT(*) AS n over the relation with the caller's WHERE. The old path rendered the clause and then cut "WHERE " off the front of the result -- a formatter's output being edited by string surgery, in a place that could not know whether the prefix was there. The clause renders it, and the condition arrives either as the WhereClause the options carry or as a bare predicate the QueryStatement wraps. Both backend classes call the builders and execute (sql, params); the async twin gets the same two calls, so the two paths have one answer between them. What changes in the rendered text, stated rather than glossed: the projection alias is dialect-quoted now (`AS `next_id``), and the COUNT alias likewise. The PK probe's parameters stay empty because the constants inline; the COUNT probe's parameters are the caller's, exactly as before. New: backend/test_internal_statements.py pins both statements -- shape, alias, that the PK probe binds nothing, that a caller's string is a parameter rather than text, and that WHERE appears exactly once. It never builds a client, so it belongs to the offline leg. The two remaining f-strings under the §16.2 grep are the CREATE / DROP SCHEMA bodies in mixins/schema.py, which are formatters -- exactly where the rule puts SQL text -- and the docstring in qualified_names.py that quotes the old spelling to explain it. Verified: feature/backend 1341 passed / 3 skipped, exit 0; the new file 8 passed, exit 0; ruff's finding set on backend/ is identical to HEAD (10 pre-existing findings, unchanged).
Core's LIKE family became words-with-keywords (f2bdf386, 4774c502): the node carries `like` / `not_like` / `ilike`, and the dialect writes the SQL keyword. That made the three members three answers this backend owes, and two of them were right by accident: the inherited formatter interpolates `expr.op.keyword`, so `ILIKE` went straight into the statement for an engine whose grammar has no such operator. The caller built a node, the renderer produced SQL, and only the server would have said no. format_like_predicate now writes the LIKE family itself: LIKE and NOT LIKE render as before, and the ILIKE member raises UnsupportedFeatureError naming the operator and the portable answer (compare LOWER(column) with LOWER(pattern)). No other backend overrides this entry, so this is the first one to draw the line where §17 draws it -- a spelling the engine lacks is refused, not silently written. ILikePredicate keeps its own separate refusal: it is a different kind making the portable LOWER() promise, and its absent formatter is what refuses it. Routing one into the other would silently rewrite SQL callers depend on, which is the reason core keeps them apart. The new test file pins both, and asserts the two refusals stay distinct -- a rejection that collapsed into a single message would mean the two promises had been merged. Verified: the new dialect file 7 passed; feature/backend 1348 passed / 3 skipped, exit 0; ruff clean on the two changed files.
Eleven rows in LEGITIMATE_NON_ROUNDTRIPS now round-trip, so the table existed only to record a defect core has fixed: * nine datetime classes lost their XML channel because DateTimeField and IntervalUnit were (str, Enum) without __str__ = str.__str__, so the wire carried 'DateTimeField.YEAR' and the constructor's normaliser refused it; * ColumnGeneratedConstraint and ForeignKeyConstraint lost all three channels because GeneratedColumnType and ReferentialAction were bare Enum, which the writer cannot name at all. Core's 3e23158e gave both families the str base and __str__, and the matrix now classifies all eleven classes as rendered. Deleting them leaves two real entries: CreateTableCloneStatement, whose constructor refuses the word its own (str, Enum) mode encodes, and CreateViewStatement, whose ViewOptions instance sits in a parameter slot the codec has no kind for. Neither is an encoding omission, so both stay, and the two integrity tests still have entries to assert instead of being deleted. The comment above the table now records both retired causes and says what the survivors are, rather than describing a root cause that no longer exists.
The dialect's surface is two kinds of member: capability switches (``supports_*()``) and formatters (``format_*(self, expr)``). This change makes BigQuery's code match that. **A string is escaped by the dialect that renders it.** The escape policy belongs to the dialect, because SQL dialects differ in how a string literal represents a quote -- BigQuery spells a quote as a backslash-quote. The base class's private copy is gone; each dialect overrides the public one. **The backend's own answers are the effective ones.** BigQuery's mixin precedes the core mixin it overrides, and a test holds that in place. **The mixin list says what it is for.** The base-list convention is written down where the list is.
Core renamed the four statement-shaped nodes it still called *Expression: TransactionExpression -> TransactionStatement SetTransactionExpression -> SetTransactionStatement SQLiteAttachExpression -> SQLiteAttachStatement SQLiteDetachExpression -> SQLiteDetachStatement This repository refers to them in its dialect signatures and its tests, so the same names follow here. Nothing changes but the names.
BigQuery's own statement nodes already named their classes, kinds and formatters alike, so this only tracks core: the datetime nodes are ``Datetime*`` because datetime is one word, the introspection nodes are ``*QueryStatement`` because they render a SELECT, and SET TRANSACTION's formatter is ``format_set_transaction_statement`` so the name it answers to matches the kind it renders.
Core moved ten cross-package helpers out from behind a leading underscore: they are called from this backend, so the underscore was claiming a scope the names never had. Each old name had exactly one new name, checked against every public definition and call site in the ten repositories first, so nothing is shadowed by the rename.
…CHEMA Add BigQuery introspection end to end: the statements, the dialect that spells them, the sync/async introspectors behind `backend.introspector`, a CLI over them, and a runnable example. No BigQuery instance was reachable while writing any of it; every statement shape is read from the documentation, and the URLs are cited beside the class that rests on them (information-schema-tables, -columns, -views, -table-constraints, -key-column-usage, -datasets-schemata, -table-options, -partitions, the INFORMATION_SCHEMA intro for the dataset/region/project qualifiers, and the system-variables page for @@project_id). What an interrupted run had left here built each question as string- concatenated SQL (`return "SELECT ... FROM " + view + ...`), which puts the backtick quoting of a `project.dataset` path in code that never sees the dialect and restates it by hand -- a project ID with a backtick would produce SQL the server refuses. That SQL is gone: every question is now a statement object in expression/introspection.py, rendered by format_* methods on BigQueryIntrospectionMixin, which assemble a core QueryStatement from Column / NamedRelationRef / ComparisonPredicate / InPredicate / LogicalPredicate / OrderByClause nodes and render it through core's own format_query_statement. The introspector only resolves *which* dataset and project a question is about, runs the rendered text, and parses the rows. Dataset is BigQuery's schema level: the core `schema` argument carries the dataset ID, and BigQueryNamespaceMixin now admits the one documented project-only path (`[PROJECT_ID.]INFORMATION_SCHEMA.SCHEMATA`) while still refusing a bare project above a relation. Core's own introspection statement names are answered too -- as adapters onto the BigQuery statements -- because declaring a capability the core entry cannot reach is worse than declining it. Foreign keys and triggers refuse with the reason: the catalog reports that a foreign key exists without naming the referenced relation, and BigQuery has no triggers. Primary keys are reported as IndexInfo with IndexType UNKNOWN, because there is no index structure behind them. CLI: tables / views / table / columns / datasets, in table/json/csv/tsv, through backend.introspector; --dataset is the schema argument, which is BigQuery's dataset. Example: prints datasets/tables/columns and shows the cache as cold vs warm executor question counts; without credentials it prints the exact env vars to set and exits 2, fabricating nothing. Tests: 74, none of them skipping, none of them touching a server -- SQL shape per builder, row->dataclass parsing fed the documented column names, cold vs warm executor question counts on both backends, and CLI dispatch with a mocked backend. The round-trip matrix and the kind-identity guard are updated for the new expression classes.
The introspect subcommand accepted a list of types and nothing in the tree said what each one reads or where its answer comes from. This adds a README to each backend's cli package: the questions with the catalogue each reads, the options, the notes that keep an answer from being misread (estimate lengths, upper-casing, version gates, capabilities that are refused rather than empty), and examples. The tables are generated from the code, not from memory -- every entry was checked against the statement it dispatches to.
The introspection CLI took a --schema flag on backends that have no schema. MySQL, MariaDB and ClickHouse have exactly one namespace level and call it a*database; Firebird has a single unnamed namespace and qualifies nothing above the*table. Asking those engines for a --schema named a level they do not have, and*the flag's help text then had to explain that the value meant something else. The flag is gone from those four. What is left is the flag the connection*already takes, which for MySQL, MariaDB and ClickHouse is --database -- so the*run connects to a database and asks about that database, which is the whole*answer when there is only one level. list_datasets() is the way to enumerate*databases; the run does not need a flag to ask about a different one. BigQuery had both --schema and --dataset naming the same level, with a*fallback order between them. One flag now, --dataset, which the connection*takes. The flags are removed rather than aliased: an alias lets a caller keep naming a*level that does not exist, which is the confusion this removes. The tests that*pinned the old flags now pin that they are gone.
A dataset is BigQuery's namespace level above a table and there is no schema under it, so this is the only namespace capability the backend has. Named `supports_dataset_introspection` so the CLI can offer `introspect datasets` and a caller can ask the question rather than try it.
…ract The row parsers are this backend's implementation of the base introspection contract: the base declares them and every backend must supply them, so the leading underscore described a visibility the code never had. It is also how the next private helper gets justified, which is what this branch is trying to stop. default_schema becomes a property rather than a method, beside dialect and cache: it states a fact about the database, it does not perform an action. No aliases: the old names are gone. BigQuery side: twelve row parsers -- datasets, table options and partitions among them -- plus default_schema and its two call sites. The parsing test follows.
"table" printed two answers for one name -- the columns and the primary key -- as two JSON arrays back to back. Those are two catalogue questions: the column listing, which ``columns`` already answers, and the key an index listing reports. The command, its help entry, and the branch in ``_report`` that printed both go with it. The registered names and the introspection list are now the same four: tables, views, columns, datasets.
supports_index_introspection() answered True while INFORMATION_SCHEMA has no index view and the CLI lists no indexes type, so the capability advertised a question the backend cannot ask. It now answers False and the INDEX scope is no longer reported; the primary-key constraint read stays available for get_primary_key(). The TABLES map also folds MATERIALIZED VIEW onto VIEW, erasing the one distinction the catalog draws now that core has TableType.MATERIALIZED_VIEW. Map it to that member, and keep VIEW and MATERIALIZED_VIEW filters separate while include_views=False still excludes both.
`fix/column-constraint-gates` was squashed into `release/v1.0.0.dev30` as `57439841`, so the branch reference this workflow used to resolve core through no longer names anything ahead of the release. Pointing at the release branch makes each run test what a user installs. The testsuite stays on `fix/column-constraint-gates` on purpose: its own branch carries the two `pytest_compat` shims that keep pytest 9.2 working (the benchmark fixture's `_nodeid` read and xdist's loadgroup suffix), and those are not on the testsuite's release branch yet. That line is repointed when the testsuite merges.
Named after the pull request once it had a number, which is the order the fragment check requires: 4 entries, one per kind of change.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## release/v1.0.0.dev1 #5 +/- ##
========================================================
+ Coverage 56.39% 68.20% +11.80%
========================================================
Files 31 44 +13
Lines 1844 2812 +968
Branches 288 373 +85
========================================================
+ Hits 1040 1918 +878
- Misses 776 847 +71
- Partials 28 47 +19
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Lands
fix/column-constraint-gatesonrelease/v1.0.0.dev1.Scope up front: this branch carries the whole cycle started on 2026-10-11 as it reaches this backend — the statement/expression refactor, the typed literals, and the introspection pass at the tip — not only the last few days of fixes. 34 commits, 5 of them marked breaking.
Headline changes:
INFORMATION_SCHEMA, and the dataset family is declared (feat(introspection): answer the metadata questions from INFORMATION_SCHEMA,feat(introspection): declare BigQuery's dataset family).fix(bigquery): stop claiming an index catalogue that does not exist)._TYPE_CAPABILITIES(feat(cli)!: the introspect table command is gone).refactor(bigquery): name the catalogue questions as statement factories).Type of change
feat: A new featurerefactor: Internal restructuring with no intended behaviour changetest: Adding missing tests or correcting existing testsdocs: Documentation onlyperf/chore/ci/build/revert(not relevant)Breaking Change
Impact on End-Users:
introspect tableno longer exists; ask forcolumns,indexesandforeign-keysseparately.indexesno longer advertises a catalogue this backend cannot answer.Impact on Backend Developers (Custom Implementations):
parse_*lost its leading underscores.Related Repositories/Packages
python-activerecord57439841pytest_compatpython-activerecord-testsuite1991449python-activerecord-bigqueryea4588cCore is merged (
rhosocial/python-activerecord#127, squashed as57439841); this branch's CI now installs core fromrelease/v1.0.0.dev30. The test suite is still referenced by branch, because itspytest_compatshims are not on its release branch yet; that reference is repointed when the test suite merges.Testing
Test Plan: The full matrix ran green on this exact head (
ea4588c) as run41092238591— every Python version and every service container the backend tests against. A local full run is skipped deliberately: the organisation caps a single task at 180 seconds, and the CI result on this SHA is the stronger evidence. The offline CLI tests (parser surface, NAME rules, capability gate) run in under a second and are part of that run.Checklist
README.md, the CLI guide).format_*).Additional Notes