Backport #31654 and #32214 to 1.13: Databricks DLT SQL lineage and self-referencing lineage - #32396
Merged
Merged
Conversation
Contributor
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
github-actions
Bot
added
Ingestion
safe to test
Contributor
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
Contributor
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
Contributor
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
…m lineage tables (#32214) * Fixes #32206: stop linking a table to itself from the Databricks system lineage tables system.access.table_lineage records table access rather than derivation, so a streaming or CDC write legitimately names its target table as its own source. The Databricks Pipeline and Unity Catalog connectors both passed those rows straight through, producing an edge from a table to itself that renders as a loop and carries no information. Skip rows where source and target are the same table, matching the guard the SQL DLT parser already applies. * fix(unitycatalog): skip self-referencing rows in the column lineage cache too The table pair they belong to is dropped, so the cached columns can never be read. They only grow column_lineage_map and inflate the cached-lineage counts in the log.
#31654) * Fixes 31278: Extract lineage from SQL-defined Databricks DLT pipelines SQL DLT transformations parsed to nothing because kafka_parser only understood the Python DLT API, so those pipelines produced no lineage. Parse CREATE [OR REFRESH] MATERIALIZED VIEW / STREAMING TABLE through the existing LineageParser, resolve qualified upstreams against their own catalog and schema, read `file` libraries, and recurse into workspace subdirectories. * refactor(databricks): dispatch DLT parsing through a language registry Adding SQL support as a fallback inside extract_dlt_table_dependencies left an if/elif chain that dispatched on failure rather than language, against the registry convention in docs/design-patterns.md. Introduce dlt_parsers.py with a DltSourceParser protocol and registry. Python and SQL each declare handles/extract and an explicit priority, so a Python notebook embedding SQL strings is not misread and a new language needs no dispatch edit. Move the parse-result dataclasses to models.py and reduce kafka_parser.py to Kafka configuration. Fix three Databricks SQL shapes found while probing real constructs: APPLY CHANGES INTO lost its source, the LIVE. prefix was resolved as a schema, and table-valued functions were returned as tables. Also use the previously dead get_pipeline_libraries helper instead of a duplicated inline copy. * fix(databricks): address PR review on DLT source discovery Glob includes were only normalised for the `/**` form, so `*.sql` and `**/*.sql` kept their wildcard and were exported as literal paths, missing every transformation. Reduce any include to the directory before its first wildcard, always ending in a slash. Stripping the LIVE. prefix could collapse two references onto one dataset, so de-duplicate upstreams while preserving order. Drop table-valued functions only when the statement invokes them, so a dataset named `range` still resolves. Strip whitespace and stray dots when qualifying a name. Remove the dead DLT_TABLE_PATTERN and the unused get_pipeline_libraries client parameter. * refactor(databricks): model DLT references and simplify parser dispatch _qualify_dlt_table_name returned a bare (catalog, schema, table) tuple, so callers had to know what each position meant. Return a DLTTableReference instead, alongside the other connector models. Replace the parser registry, the Protocol priority field and the per-call sort with an explicit ordered DLT_PARSERS list. Two implementations did not justify the indirection, and the previous approach also misused enum_register, which everywhere else keys functions by an enum value rather than classes by an ad-hoc string. Order is now stated where the list is defined, and a test asserts it rather than assuming it. Bound the entity caches with LRUCache, which CLAUDE.md requires and the sibling Databricks ownership source already uses. Typing the cache surfaced that fqn.build can return None and the old code cached under a None key, so collapse the two duplicated cache blocks into a single _lookup_table helper that guards it once. Build the source through the real create() in tests instead of assigning its internals. The hand-assigned caches had already drifted from production once without any test failing. Cache bounds, eviction, miss caching and the None guard are now asserted against the real object. * fix(databricks): select only the files a DLT glob actually matches glob_base_directory reduced an include to its base directory and the caller then expanded that whole tree, so a selective include such as /transformations/*.sql pulled in unrelated files and attributed their lineage to the pipeline. That is worse than the wildcard bug it replaced, because the edges are wrong rather than absent. Model a library entry as DLTLibrarySource, carrying the directory to list and the pattern its contents must match. Recurse only for **, and check every candidate with glob_matches, where * stays inside a path segment and ** spans them. fnmatch is unsuitable because its * also crosses /. Tests now assert which files an include selects rather than which directory it reduces to, which is what the earlier assertion missed. * fix(databricks): keep library entries modelled and treat ? as a wildcard get_pipeline_libraries returns DLTLibrarySource, but the spec.configuration and spec.development source_path fallback still appended a raw string, so iterating the list hit AttributeError on library.is_directory and lineage died for pipelines that use those fields. glob_base_directory only looked for *, while glob_matches also honours ?, so an include such as /transformations/file_?.sql was treated as a concrete path and sent to the export API literally. Both now share GLOB_WILDCARDS so they cannot disagree again. * fix(databricks): expand a pattern-less library directory in full A configuration or development source_path produces a library with no pattern, and treating that as non-recursive skipped every nested transformation. No pattern means no filter, so the whole tree is in scope. An include that names a bare directory was also stored as its own pattern, which then matched nothing and dropped the directory entirely. Only attach a pattern when the include actually contains a wildcard. Cover every spec.libraries shape end to end so the reduction and the matching cannot drift apart again. * fix(databricks): state directory-ness instead of inferring it from the path is_directory was read off a trailing slash, so a source_path that Databricks spells without one was exported as a file and its transformations never listed. The producer knows what each entry is, so it now says so: notebook and file entries are files, a wildcard include is always a directory, and a source_path is normalised to a directory. is_recursive only looked for **, so a wildcard in a directory segment such as /tx/2024_?/file.sql reduced to /tx/ and then refused to descend, selecting nothing. Decide from the part of the pattern below the base directory, where either ** or a / means traversal is required. * fix(databricks): let the workspace settle an ambiguous glob include A glob include with neither a wildcard nor a trailing slash may name a file or a directory, and the string cannot tell them apart: assuming a file leaves a directory unlisted, assuming a directory swallows a concrete .sql. Model the gap instead, with is_directory None meaning the spec did not say, and resolve it by listing the path. Anything that lists is a directory, anything that does not is read as a single source. * fix(databricks): read the legacy and private DLT create forms CREATE LIVE TABLE, CREATE STREAMING LIVE TABLE and the PRIVATE modifier are spellings DLT still accepts but the query parser cannot read. It falls back to a bare command and reports no tables, so the dataset was dropped with neither an error nor a warning. Rewrite them to the modern equivalent before parsing, anchored to the CREATE clause so a dataset whose own name contains "live" is left alone. * refactor(databricks): drop the glob machinery Databricks cannot produce The pipelines API accepts only an exact file, a directory, or a directory with a trailing **, and rejects every other wildcard form outright. The pattern matcher, the wildcard set and the recursive flag were therefore unreachable, along with the tests asserting on them. A library is now just a file or a directory taken in full. * fix(databricks): describe library discovery as it behaves and lift the depth cap The get_pipeline_libraries docstring still promised pattern filtering that no longer exists, and the expansion cap of 5 silently dropped transformations nested deeper than that, which costs the lineage depending on them. Correct the docstring and raise the cap to 20. A directory that is not there costs nothing to look for. * fix(databricks): use the real lookup cache in the self-referencing lineage test The test stubbed _table_lookup_cache with a plain dict, which satisfied main's implementation but not this branch's LRUCache. The lookup raised, the exception was swallowed, and the two tests expecting an edge failed once main was merged in. * fix(databricks): read a null libraries value and quieten per-file logging The libraries key can be present and null, and testing for the key rather than reading it made len() raise before the source paths were collected, which cost the pipeline its lineage. Per-path discovery also logged at INFO, which buries the run summary now that expansion is recursive.
…e workspace root (#32404) * Fixes #32397: stop a wildcard include with no directory reducing to the workspace root glob_base_directory truncates an include at its first wildcard and falls back to the parent directory. With nothing before the wildcard there is no parent, so "**" and "*" reduced to "/" and the expansion walked the entire workspace to max_depth. Return such an include unchanged so it resolves to nothing, which is far cheaper than listing everything. * fix(databricks): leave a first-segment wildcard include alone as well A base of "/" took the early return and handed back the workspace root, so "/*.sql" still reduced to "/" and expansion listed everything. The docstring also promised a real directory fallback that no longer holds for these forms. Return the include unchanged whenever the only directory above it is the root.
1.13 checks imports with isort rather than ruff, and the multi-line form the ported file carries is not what it wants.
mohittilala
force-pushed
the
backport/databricks-dlt-lineage-1.13
branch
from
September 2, 2026 14:56
d402825 to
6ac8f72
Compare
basedpyright reads the ported files against 1.13's generated models rather than main's, and flagged eleven errors the pristine connector did not have. Left as is they would fail Run Static Checks for every later PR on this branch, since the check covers all of src. Guard the optional pipeline and run state, take table ids without unwrapping root, and pass Either its left explicitly where the nesting depth defeats the generic default.
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
Code Review ✅ Approved 1 resolved / 1 findingsBackports Databricks DLT SQL lineage extraction and self-referencing lineage fixes from main to 1.13. SQL-defined DLT pipelines now yield lineage, and tables no longer link to themselves via ✅ 1 resolved✅ Edge Case: Pipeline-level executionStatus still dereferences run.state unguarded
OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
ulixius9
approved these changes
Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes:
Backport of two merged Databricks lineage fixes to
1.13.CREATE LIVE TABLE/STREAMING LIVE TABLE/PRIVATEforms that matched the pattern but produced nothing, and removes glob handling the pipelines API cannot produce.system.access.table_lineagecan return rows whose source and target are the same table. Both the Databricks Pipeline and Unity Catalog connectors turned those into an edge from a table to itself.The four
databrickspipelinefiles and both new test files are byte-identical tomain, so future backports of these files apply cleanly.unitycatalog/lineage.pyis the exception and keeps 1.13's own structure. That file has genuinely diverged onmainbeyond formatting (TYPE_CHECKING, different connection imports, a restructuredcreate()), so taking main's copy would drag unrelated refactoring into 1.13. Both self-reference guards are added there verbatim frommain.Formatting note: 1.13 runs
blackandisortwhilemainrunsruff, so the ported files carry ruff-era# noqacomments and 120 column lines.black --checkwill flag them. This is deliberate, to keep the hunks matchingmain. Henceskip-pr-checks.Type of change:
High-level design:
N/A, backport with no new design.
Tests:
Use cases covered
Same as the source PRs. Verified on this branch rather than assumed:
main.main, covering every library shape the pipelines API accepts (exact file, bare directory, trailing slash,**, mixed entries).Not verified on this branch: the OpenMetadata edge count. The local server is 2.0.0 and the 1.13 ingestion client refuses to connect (
Server version is 2.0.0 vs. Client version 1.13.4.1), which is a version guard rather than anything in this change. The 2.0 backport, running byte-identical connector code against identical parse results, wrote the expected 34 edges.Backend integration tests
Ingestion integration tests
Playwright (UI) tests
Manual testing performed
mainto confirm the result matches, with theunitycatalog/lineage.pyexception noted above.UI screen recording / screenshots:
Not applicable.
Checklist: