fix(ingestion): resolve OpenLineage column-lineage FQNs by real store… - #31651
fix(ingestion): resolve OpenLineage column-lineage FQNs by real store…#31651romsharon98 wants to merge 2 commits into
Conversation
…d column case _get_column_lineage unconditionally lowercased OpenLineage field names when building column FQNs. Destinations whose real stored columns are uppercase (e.g. Snowflake) got column FQNs matching no real column, so the server silently dropped the columnsLineage entry even though the client sent a populated one. Now resolves the real column name case-insensitively from the resolved Table entity instead of blindly lowercasing. Also extends the backtick-stripping from open-metadata#30240 to _parse_slash_table_name (Azure Kusto naming), which shares the same unstripped-backtick pattern that was fixed for _parse_glue_table_name but not applied here. Verified end-to-end against a live OpenMetadata 1.13.1 instance: after this fix, a synthetic OpenLineage event for a real silver (Glue, lowercase columns) -> gold (Snowflake, uppercase columns) edge produced correct columnsLineage with real stored casing, where it previously landed empty.
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
Code Review ✅ ApprovedResolves OpenLineage column-lineage FQNs case-insensitively using real stored table column names instead of unconditionally lowercasing them, and extends backtick-stripping to Azure Kusto table names. No issues found. OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| for column in getattr(table, "columns", None) or []: | ||
| if column.name.root.lower() == lower: | ||
| return column.name.root |
There was a problem hiding this comment.
When an OpenLineage field's case differs from the stored column, the Table lookups do not request the non-default columns field, so _match_column_name falls back to the lowercased event field and the server can continue dropping the resulting nonexistent column FQN.
1 participant
fix(ingestion): resolve OpenLineage column-lineage FQNs by real stored column case
_get_column_lineage unconditionally lowercased OpenLineage field names when
building column FQNs. Destinations whose real stored columns are uppercase
(e.g. Snowflake) got column FQNs matching no real column, so the server
silently dropped the columnsLineage entry even though the client sent a
populated one. Now resolves the real column name case-insensitively from the
resolved Table entity instead of blindly lowercasing.
Also extends the backtick-stripping from #30240 to _parse_slash_table_name
(Azure Kusto naming), which shares the same unstripped-backtick pattern that
was fixed for _parse_glue_table_name but not applied here.
Verified end-to-end against a live OpenMetadata 1.13.1 instance: after this
fix, a synthetic OpenLineage event for a real silver (Glue, lowercase
columns) -> gold (Snowflake, uppercase columns) edge produced correct
columnsLineage with real stored casing, where it previously landed empty.
Describe your changes:
Fixes #
I worked on ... because ...
Type of change:
High-level design:
N/A — small change.
Tests:
Use cases covered
Unit tests
Backend integration tests
Ingestion integration tests
Playwright (UI) tests
Manual testing performed
UI screen recording / screenshots:
Not applicable.
Checklist:
Fixes <issue-number>: <short explanation>Fixes #<issue-number>above.Greptile Summary
This PR updates OpenLineage parsing and column-lineage FQN construction to strip Azure Kusto backticks and preserve the casing of stored columns.
Confidence Score: 4/5
The PR should not merge until column-bearing Table entities are supplied to the new matcher, otherwise the reported uppercase-column failure remains reachable.
The matcher correctly preserves stored casing only when
table.columnsis populated, while its production lookup path does not request that non-default field and may return an earlier field-less cached entity.Files Needing Attention: ingestion/src/metadata/ingestion/source/pipeline/openlineage/metadata.py
Important Files Changed
Reviews (1): Last reviewed commit: "Merge branch 'main' into fix/openlineage..." | Re-trigger Greptile