Fixes 32397: stop a wildcard include with no directory reducing to the workspace root - #32404
Merged
Merged
Conversation
…he workspace root glob_base_directory truncates an include at its first wildcard and falls back to the parent directory. With nothing before the wildcard there is no parent, so "**" and "*" reduced to "/" and the expansion walked the entire workspace to max_depth. Return such an include unchanged so it resolves to nothing, which is far cheaper than listing everything.
github-actions
Bot
added
the
safe to test
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens Databricks DLT pipeline library path handling in ingestion by preventing “bare wildcard” glob includes from collapsing to the workspace root (/), which could otherwise trigger an expensive full-workspace traversal during library expansion.
Changes:
- Updates
glob_base_directory()to avoid reducing wildcard-only includes (e.g.**,*,*.sql) to/. - Adds unit tests to ensure bare wildcard includes remain unchanged and that workspace expansion never lists from
/for those cases.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| ingestion/src/metadata/ingestion/source/pipeline/databrickspipeline/kafka_parser.py | Adjusts base-directory reduction logic to avoid collapsing bare wildcard includes to the workspace root. |
| ingestion/tests/unit/topology/pipeline/test_databricks_dlt_sql_lineage.py | Adds regression tests covering wildcard-with-no-directory behavior and ensuring / is never expanded for that scenario. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
✅ Playwright Results — workflow succeededValidated commit ✅ 109 passed · ❌ 0 failed · 🟡 1 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky PerformanceBlocking targets: ✅ met · Optimization targets: 🟡 in progress Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting. 🕒 Full workflow signal wall (to summary) 51m 46s ⏱️ Max setup 4m 59s · max shard execution 12m 57s · max shard-job elapsed before upload 20m 7s · reporting 5s 🌐 218.28 requests/attempt · 1.78 app boots/UI scenario · 0.00% common-shard skew Optimization targets still in progress:
🟡 1 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
A base of "/" took the early return and handed back the workspace root, so "/*.sql" still reduced to "/" and expansion listed everything. The docstring also promised a real directory fallback that no longer holds for these forms. Return the include unchanged whenever the only directory above it is the root.
Code Review ✅ ApprovedFixes a path reduction bug where OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The change is narrowly scoped, preserves existing behavior for valid directory includes, and is covered by targeted unit tests for the regression scenario.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
harshsoni2024
approved these changes
Sep 2, 2026
mohittilala
added a commit
that referenced
this pull request
Sep 2, 2026
…e workspace root (#32404) * Fixes #32397: stop a wildcard include with no directory reducing to the workspace root glob_base_directory truncates an include at its first wildcard and falls back to the parent directory. With nothing before the wildcard there is no parent, so "**" and "*" reduced to "/" and the expansion walked the entire workspace to max_depth. Return such an include unchanged so it resolves to nothing, which is far cheaper than listing everything. * fix(databricks): leave a first-segment wildcard include alone as well A base of "/" took the early return and handed back the workspace root, so "/*.sql" still reduced to "/" and expansion listed everything. The docstring also promised a real directory fallback that no longer holds for these forms. Return the include unchanged whenever the only directory above it is the root.
mohittilala
added a commit
that referenced
this pull request
Sep 2, 2026
…e workspace root (#32404) * Fixes #32397: stop a wildcard include with no directory reducing to the workspace root glob_base_directory truncates an include at its first wildcard and falls back to the parent directory. With nothing before the wildcard there is no parent, so "**" and "*" reduced to "/" and the expansion walked the entire workspace to max_depth. Return such an include unchanged so it resolves to nothing, which is far cheaper than listing everything. * fix(databricks): leave a first-segment wildcard include alone as well A base of "/" took the early return and handed back the workspace root, so "/*.sql" still reduced to "/" and expansion listed everything. The docstring also promised a real directory fallback that no longer holds for these forms. Return the include unchanged whenever the only directory above it is the root.
mohittilala
added a commit
that referenced
this pull request
Sep 2, 2026
…e workspace root (#32404) * Fixes #32397: stop a wildcard include with no directory reducing to the workspace root glob_base_directory truncates an include at its first wildcard and falls back to the parent directory. With nothing before the wildcard there is no parent, so "**" and "*" reduced to "/" and the expansion walked the entire workspace to max_depth. Return such an include unchanged so it resolves to nothing, which is far cheaper than listing everything. * fix(databricks): leave a first-segment wildcard include alone as well A base of "/" took the early return and handed back the workspace root, so "/*.sql" still reduced to "/" and expansion listed everything. The docstring also promised a real directory fallback that no longer holds for these forms. Return the include unchanged whenever the only directory above it is the root.
Rohit0301
pushed a commit
that referenced
this pull request
Sep 3, 2026
…e workspace root (#32404) * Fixes #32397: stop a wildcard include with no directory reducing to the workspace root glob_base_directory truncates an include at its first wildcard and falls back to the parent directory. With nothing before the wildcard there is no parent, so "**" and "*" reduced to "/" and the expansion walked the entire workspace to max_depth. Return such an include unchanged so it resolves to nothing, which is far cheaper than listing everything. * fix(databricks): leave a first-segment wildcard include alone as well A base of "/" took the early return and handed back the workspace root, so "/*.sql" still reduced to "/" and expansion listed everything. The docstring also promised a real directory fallback that no longer holds for these forms. Return the include unchanged whenever the only directory above it is the root.
mohittilala
added a commit
that referenced
this pull request
Sep 3, 2026
…lf-referencing lineage (#32396) * Fixes 32206: stop linking a table to itself from the Databricks system lineage tables (#32214) * Fixes #32206: stop linking a table to itself from the Databricks system lineage tables system.access.table_lineage records table access rather than derivation, so a streaming or CDC write legitimately names its target table as its own source. The Databricks Pipeline and Unity Catalog connectors both passed those rows straight through, producing an edge from a table to itself that renders as a loop and carries no information. Skip rows where source and target are the same table, matching the guard the SQL DLT parser already applies. * fix(unitycatalog): skip self-referencing rows in the column lineage cache too The table pair they belong to is dropped, so the cached columns can never be read. They only grow column_lineage_map and inflate the cached-lineage counts in the log. * Fixes 31278: Extract lineage from SQL-defined Databricks DLT pipelines (#31654) * Fixes 31278: Extract lineage from SQL-defined Databricks DLT pipelines SQL DLT transformations parsed to nothing because kafka_parser only understood the Python DLT API, so those pipelines produced no lineage. Parse CREATE [OR REFRESH] MATERIALIZED VIEW / STREAMING TABLE through the existing LineageParser, resolve qualified upstreams against their own catalog and schema, read `file` libraries, and recurse into workspace subdirectories. * refactor(databricks): dispatch DLT parsing through a language registry Adding SQL support as a fallback inside extract_dlt_table_dependencies left an if/elif chain that dispatched on failure rather than language, against the registry convention in docs/design-patterns.md. Introduce dlt_parsers.py with a DltSourceParser protocol and registry. Python and SQL each declare handles/extract and an explicit priority, so a Python notebook embedding SQL strings is not misread and a new language needs no dispatch edit. Move the parse-result dataclasses to models.py and reduce kafka_parser.py to Kafka configuration. Fix three Databricks SQL shapes found while probing real constructs: APPLY CHANGES INTO lost its source, the LIVE. prefix was resolved as a schema, and table-valued functions were returned as tables. Also use the previously dead get_pipeline_libraries helper instead of a duplicated inline copy. * fix(databricks): address PR review on DLT source discovery Glob includes were only normalised for the `/**` form, so `*.sql` and `**/*.sql` kept their wildcard and were exported as literal paths, missing every transformation. Reduce any include to the directory before its first wildcard, always ending in a slash. Stripping the LIVE. prefix could collapse two references onto one dataset, so de-duplicate upstreams while preserving order. Drop table-valued functions only when the statement invokes them, so a dataset named `range` still resolves. Strip whitespace and stray dots when qualifying a name. Remove the dead DLT_TABLE_PATTERN and the unused get_pipeline_libraries client parameter. * refactor(databricks): model DLT references and simplify parser dispatch _qualify_dlt_table_name returned a bare (catalog, schema, table) tuple, so callers had to know what each position meant. Return a DLTTableReference instead, alongside the other connector models. Replace the parser registry, the Protocol priority field and the per-call sort with an explicit ordered DLT_PARSERS list. Two implementations did not justify the indirection, and the previous approach also misused enum_register, which everywhere else keys functions by an enum value rather than classes by an ad-hoc string. Order is now stated where the list is defined, and a test asserts it rather than assuming it. Bound the entity caches with LRUCache, which CLAUDE.md requires and the sibling Databricks ownership source already uses. Typing the cache surfaced that fqn.build can return None and the old code cached under a None key, so collapse the two duplicated cache blocks into a single _lookup_table helper that guards it once. Build the source through the real create() in tests instead of assigning its internals. The hand-assigned caches had already drifted from production once without any test failing. Cache bounds, eviction, miss caching and the None guard are now asserted against the real object. * fix(databricks): select only the files a DLT glob actually matches glob_base_directory reduced an include to its base directory and the caller then expanded that whole tree, so a selective include such as /transformations/*.sql pulled in unrelated files and attributed their lineage to the pipeline. That is worse than the wildcard bug it replaced, because the edges are wrong rather than absent. Model a library entry as DLTLibrarySource, carrying the directory to list and the pattern its contents must match. Recurse only for **, and check every candidate with glob_matches, where * stays inside a path segment and ** spans them. fnmatch is unsuitable because its * also crosses /. Tests now assert which files an include selects rather than which directory it reduces to, which is what the earlier assertion missed. * fix(databricks): keep library entries modelled and treat ? as a wildcard get_pipeline_libraries returns DLTLibrarySource, but the spec.configuration and spec.development source_path fallback still appended a raw string, so iterating the list hit AttributeError on library.is_directory and lineage died for pipelines that use those fields. glob_base_directory only looked for *, while glob_matches also honours ?, so an include such as /transformations/file_?.sql was treated as a concrete path and sent to the export API literally. Both now share GLOB_WILDCARDS so they cannot disagree again. * fix(databricks): expand a pattern-less library directory in full A configuration or development source_path produces a library with no pattern, and treating that as non-recursive skipped every nested transformation. No pattern means no filter, so the whole tree is in scope. An include that names a bare directory was also stored as its own pattern, which then matched nothing and dropped the directory entirely. Only attach a pattern when the include actually contains a wildcard. Cover every spec.libraries shape end to end so the reduction and the matching cannot drift apart again. * fix(databricks): state directory-ness instead of inferring it from the path is_directory was read off a trailing slash, so a source_path that Databricks spells without one was exported as a file and its transformations never listed. The producer knows what each entry is, so it now says so: notebook and file entries are files, a wildcard include is always a directory, and a source_path is normalised to a directory. is_recursive only looked for **, so a wildcard in a directory segment such as /tx/2024_?/file.sql reduced to /tx/ and then refused to descend, selecting nothing. Decide from the part of the pattern below the base directory, where either ** or a / means traversal is required. * fix(databricks): let the workspace settle an ambiguous glob include A glob include with neither a wildcard nor a trailing slash may name a file or a directory, and the string cannot tell them apart: assuming a file leaves a directory unlisted, assuming a directory swallows a concrete .sql. Model the gap instead, with is_directory None meaning the spec did not say, and resolve it by listing the path. Anything that lists is a directory, anything that does not is read as a single source. * fix(databricks): read the legacy and private DLT create forms CREATE LIVE TABLE, CREATE STREAMING LIVE TABLE and the PRIVATE modifier are spellings DLT still accepts but the query parser cannot read. It falls back to a bare command and reports no tables, so the dataset was dropped with neither an error nor a warning. Rewrite them to the modern equivalent before parsing, anchored to the CREATE clause so a dataset whose own name contains "live" is left alone. * refactor(databricks): drop the glob machinery Databricks cannot produce The pipelines API accepts only an exact file, a directory, or a directory with a trailing **, and rejects every other wildcard form outright. The pattern matcher, the wildcard set and the recursive flag were therefore unreachable, along with the tests asserting on them. A library is now just a file or a directory taken in full. * fix(databricks): describe library discovery as it behaves and lift the depth cap The get_pipeline_libraries docstring still promised pattern filtering that no longer exists, and the expansion cap of 5 silently dropped transformations nested deeper than that, which costs the lineage depending on them. Correct the docstring and raise the cap to 20. A directory that is not there costs nothing to look for. * fix(databricks): use the real lookup cache in the self-referencing lineage test The test stubbed _table_lookup_cache with a plain dict, which satisfied main's implementation but not this branch's LRUCache. The lookup raised, the exception was swallowed, and the two tests expecting an edge failed once main was merged in. * fix(databricks): read a null libraries value and quieten per-file logging The libraries key can be present and null, and testing for the key rather than reading it made len() raise before the source paths were collected, which cost the pipeline its lineage. Per-path discovery also logged at INFO, which buries the run summary now that expansion is recursive. * Empty commit to trigger fresh CI * Fixes 32397: stop a wildcard include with no directory reducing to the workspace root (#32404) * Fixes #32397: stop a wildcard include with no directory reducing to the workspace root glob_base_directory truncates an include at its first wildcard and falls back to the parent directory. With nothing before the wildcard there is no parent, so "**" and "*" reduced to "/" and the expansion walked the entire workspace to max_depth. Return such an include unchanged so it resolves to nothing, which is far cheaper than listing everything. * fix(databricks): leave a first-segment wildcard include alone as well A base of "/" took the early return and handed back the workspace root, so "/*.sql" still reduced to "/" and expansion listed everything. The docstring also promised a real directory fallback that no longer holds for these forms. Return the include unchanged whenever the only directory above it is the root. * fix(databricks): sort the SQL parser's deferred import for 1.13 1.13 checks imports with isort rather than ruff, and the multi-line form the ported file carries is not what it wants. * fix(databricks): satisfy 1.13 type checks in the ported connector basedpyright reads the ported files against 1.13's generated models rather than main's, and flagged eleven errors the pristine connector did not have. Left as is they would fail Run Static Checks for every later PR on this branch, since the check covers all of src. Guard the optional pipeline and run state, take table ids without unwrapping root, and pass Either its left explicitly where the nesting depth defeats the generic default.
mohittilala
added a commit
that referenced
this pull request
Sep 3, 2026
…f-referencing lineage (#32395) * Fixes 32206: stop linking a table to itself from the Databricks system lineage tables (#32214) * Fixes #32206: stop linking a table to itself from the Databricks system lineage tables system.access.table_lineage records table access rather than derivation, so a streaming or CDC write legitimately names its target table as its own source. The Databricks Pipeline and Unity Catalog connectors both passed those rows straight through, producing an edge from a table to itself that renders as a loop and carries no information. Skip rows where source and target are the same table, matching the guard the SQL DLT parser already applies. * fix(unitycatalog): skip self-referencing rows in the column lineage cache too The table pair they belong to is dropped, so the cached columns can never be read. They only grow column_lineage_map and inflate the cached-lineage counts in the log. * Fixes 31278: Extract lineage from SQL-defined Databricks DLT pipelines (#31654) * Fixes 31278: Extract lineage from SQL-defined Databricks DLT pipelines SQL DLT transformations parsed to nothing because kafka_parser only understood the Python DLT API, so those pipelines produced no lineage. Parse CREATE [OR REFRESH] MATERIALIZED VIEW / STREAMING TABLE through the existing LineageParser, resolve qualified upstreams against their own catalog and schema, read `file` libraries, and recurse into workspace subdirectories. * refactor(databricks): dispatch DLT parsing through a language registry Adding SQL support as a fallback inside extract_dlt_table_dependencies left an if/elif chain that dispatched on failure rather than language, against the registry convention in docs/design-patterns.md. Introduce dlt_parsers.py with a DltSourceParser protocol and registry. Python and SQL each declare handles/extract and an explicit priority, so a Python notebook embedding SQL strings is not misread and a new language needs no dispatch edit. Move the parse-result dataclasses to models.py and reduce kafka_parser.py to Kafka configuration. Fix three Databricks SQL shapes found while probing real constructs: APPLY CHANGES INTO lost its source, the LIVE. prefix was resolved as a schema, and table-valued functions were returned as tables. Also use the previously dead get_pipeline_libraries helper instead of a duplicated inline copy. * fix(databricks): address PR review on DLT source discovery Glob includes were only normalised for the `/**` form, so `*.sql` and `**/*.sql` kept their wildcard and were exported as literal paths, missing every transformation. Reduce any include to the directory before its first wildcard, always ending in a slash. Stripping the LIVE. prefix could collapse two references onto one dataset, so de-duplicate upstreams while preserving order. Drop table-valued functions only when the statement invokes them, so a dataset named `range` still resolves. Strip whitespace and stray dots when qualifying a name. Remove the dead DLT_TABLE_PATTERN and the unused get_pipeline_libraries client parameter. * refactor(databricks): model DLT references and simplify parser dispatch _qualify_dlt_table_name returned a bare (catalog, schema, table) tuple, so callers had to know what each position meant. Return a DLTTableReference instead, alongside the other connector models. Replace the parser registry, the Protocol priority field and the per-call sort with an explicit ordered DLT_PARSERS list. Two implementations did not justify the indirection, and the previous approach also misused enum_register, which everywhere else keys functions by an enum value rather than classes by an ad-hoc string. Order is now stated where the list is defined, and a test asserts it rather than assuming it. Bound the entity caches with LRUCache, which CLAUDE.md requires and the sibling Databricks ownership source already uses. Typing the cache surfaced that fqn.build can return None and the old code cached under a None key, so collapse the two duplicated cache blocks into a single _lookup_table helper that guards it once. Build the source through the real create() in tests instead of assigning its internals. The hand-assigned caches had already drifted from production once without any test failing. Cache bounds, eviction, miss caching and the None guard are now asserted against the real object. * fix(databricks): select only the files a DLT glob actually matches glob_base_directory reduced an include to its base directory and the caller then expanded that whole tree, so a selective include such as /transformations/*.sql pulled in unrelated files and attributed their lineage to the pipeline. That is worse than the wildcard bug it replaced, because the edges are wrong rather than absent. Model a library entry as DLTLibrarySource, carrying the directory to list and the pattern its contents must match. Recurse only for **, and check every candidate with glob_matches, where * stays inside a path segment and ** spans them. fnmatch is unsuitable because its * also crosses /. Tests now assert which files an include selects rather than which directory it reduces to, which is what the earlier assertion missed. * fix(databricks): keep library entries modelled and treat ? as a wildcard get_pipeline_libraries returns DLTLibrarySource, but the spec.configuration and spec.development source_path fallback still appended a raw string, so iterating the list hit AttributeError on library.is_directory and lineage died for pipelines that use those fields. glob_base_directory only looked for *, while glob_matches also honours ?, so an include such as /transformations/file_?.sql was treated as a concrete path and sent to the export API literally. Both now share GLOB_WILDCARDS so they cannot disagree again. * fix(databricks): expand a pattern-less library directory in full A configuration or development source_path produces a library with no pattern, and treating that as non-recursive skipped every nested transformation. No pattern means no filter, so the whole tree is in scope. An include that names a bare directory was also stored as its own pattern, which then matched nothing and dropped the directory entirely. Only attach a pattern when the include actually contains a wildcard. Cover every spec.libraries shape end to end so the reduction and the matching cannot drift apart again. * fix(databricks): state directory-ness instead of inferring it from the path is_directory was read off a trailing slash, so a source_path that Databricks spells without one was exported as a file and its transformations never listed. The producer knows what each entry is, so it now says so: notebook and file entries are files, a wildcard include is always a directory, and a source_path is normalised to a directory. is_recursive only looked for **, so a wildcard in a directory segment such as /tx/2024_?/file.sql reduced to /tx/ and then refused to descend, selecting nothing. Decide from the part of the pattern below the base directory, where either ** or a / means traversal is required. * fix(databricks): let the workspace settle an ambiguous glob include A glob include with neither a wildcard nor a trailing slash may name a file or a directory, and the string cannot tell them apart: assuming a file leaves a directory unlisted, assuming a directory swallows a concrete .sql. Model the gap instead, with is_directory None meaning the spec did not say, and resolve it by listing the path. Anything that lists is a directory, anything that does not is read as a single source. * fix(databricks): read the legacy and private DLT create forms CREATE LIVE TABLE, CREATE STREAMING LIVE TABLE and the PRIVATE modifier are spellings DLT still accepts but the query parser cannot read. It falls back to a bare command and reports no tables, so the dataset was dropped with neither an error nor a warning. Rewrite them to the modern equivalent before parsing, anchored to the CREATE clause so a dataset whose own name contains "live" is left alone. * refactor(databricks): drop the glob machinery Databricks cannot produce The pipelines API accepts only an exact file, a directory, or a directory with a trailing **, and rejects every other wildcard form outright. The pattern matcher, the wildcard set and the recursive flag were therefore unreachable, along with the tests asserting on them. A library is now just a file or a directory taken in full. * fix(databricks): describe library discovery as it behaves and lift the depth cap The get_pipeline_libraries docstring still promised pattern filtering that no longer exists, and the expansion cap of 5 silently dropped transformations nested deeper than that, which costs the lineage depending on them. Correct the docstring and raise the cap to 20. A directory that is not there costs nothing to look for. * fix(databricks): use the real lookup cache in the self-referencing lineage test The test stubbed _table_lookup_cache with a plain dict, which satisfied main's implementation but not this branch's LRUCache. The lookup raised, the exception was swallowed, and the two tests expecting an edge failed once main was merged in. * fix(databricks): read a null libraries value and quieten per-file logging The libraries key can be present and null, and testing for the key rather than reading it made len() raise before the source paths were collected, which cost the pipeline its lineage. Per-path discovery also logged at INFO, which buries the run summary now that expansion is recursive. * Fixes 32397: stop a wildcard include with no directory reducing to the workspace root (#32404) * Fixes #32397: stop a wildcard include with no directory reducing to the workspace root glob_base_directory truncates an include at its first wildcard and falls back to the parent directory. With nothing before the wildcard there is no parent, so "**" and "*" reduced to "/" and the expansion walked the entire workspace to max_depth. Return such an include unchanged so it resolves to nothing, which is far cheaper than listing everything. * fix(databricks): leave a first-segment wildcard include alone as well A base of "/" took the early return and handed back the workspace root, so "/*.sql" still reduced to "/" and expansion listed everything. The docstring also promised a real directory fallback that no longer holds for these forms. Return the include unchanged whenever the only directory above it is the root. * fix(databricks): suppress PLC0415 on the SQL parser's deferred imports 2.0 enables the pylint convention rules that main does not, so the imports kept inside the SQL parser to avoid paying for the lineage stack up front fail its checkstyle. Suppress them the way the connector already does elsewhere.
shrabantipaul-collate
pushed a commit
to PRADDZY/OpenMetadata
that referenced
this pull request
Sep 3, 2026
…e workspace root (open-metadata#32404) * Fixes open-metadata#32397: stop a wildcard include with no directory reducing to the workspace root glob_base_directory truncates an include at its first wildcard and falls back to the parent directory. With nothing before the wildcard there is no parent, so "**" and "*" reduced to "/" and the expansion walked the entire workspace to max_depth. Return such an include unchanged so it resolves to nothing, which is far cheaper than listing everything. * fix(databricks): leave a first-segment wildcard include alone as well A base of "/" took the early return and handed back the workspace root, so "/*.sql" still reduced to "/" and expansion listed everything. The docstring also promised a real directory fallback that no longer holds for these forms. Return the include unchanged whenever the only directory above it is the root.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes:
Fixes #32397
glob_base_directoryreduces aspec.librariesglob include to the directory the caller should list. It truncates at the first wildcard and falls back to the parent directory when the truncation does not already end in/.With nothing before the wildcard there is no parent, so the fallback landed on the workspace root:
_expand_workspace_directorythen walked from/tomax_depth, listing every notebook and file in the workspace instead of skipping an include it cannot resolve.Such an include is now returned unchanged, so it resolves to nothing. That is far cheaper than listing everything, and an include that carries a directory still reduces exactly as before.
Defensive only. The pipelines API rejects
*,?, and a bare**, so a working pipeline cannot produce one. Reported by an automated reviewer on the 2.0 backport of #31654.Type of change:
High-level design:
N/A, small change.
Tests:
Use cases covered
**,*,*.sqland?.sqlare returned unchanged rather than reduced to/./tx/**,/tx/staging*and/tx/still reduce to/tx/./tx/one.sqlis still left alone._expand_workspace_directorynever lists/for a bare wildcard include.Unit tests
ingestion/tests/unit/topology/pipeline/test_databricks_dlt_sql_lineage.py(TestWildcardWithNoDirectory, 4 tests)test_a_bare_wildcard_is_returned_unchangedandtest_the_workspace_root_is_never_expanded, while the two control tests covering directory includes and concrete paths still pass.Backend integration tests
Ingestion integration tests
Playwright (UI) tests
Manual testing performed
mainreturns/for**,*and*.sql.UI screen recording / screenshots:
Not applicable.
Checklist:
Fixes <issue-number>: <short explanation>Fixes #<issue-number>above.