fix(datasource): filter Hive partition metadata from table fields - #1367
Merged
XiaJunjie2020 merged 2 commits intoSep 15, 2026
Merged
Conversation
Ignore blank names and DESCRIBE metadata headers. Retain partition columns once in their original order, preserve valid hash-prefixed column names, and keep support for extra driver columns. Refs dataease#1250.
Add eight regression cases for partition metadata, duplicate fields, blank names, valid hash-prefixed columns, empty results, and extra driver columns. Execute the production get_fields and ColumnSchema definitions with a mocked Hive cursor. Validation: python -m pytest backend/tests -q (11 passed).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1250.
Hive
DESCRIBEreturns blank rows and section headers such as# Partition Informationand# col_name, and can repeat partition columns. These rows currently become table fields, causing data previews to reference metadata instead of real columns.Filter blank names and metadata headers in the Hive field-discovery branch, and deduplicate fields while preserving the first occurrence and original order. Keep real partition columns, valid hash-prefixed column names, and compatibility with extra driver columns.
Validation (Python 3.11):
python -m pytest backend/tests -q: 11 passed, including 8 Hive regression cases.ruff check backend/tests/test_hive_fields.pyandgit diff --check: passed.tests backend/tests): 117 passed, 3 skipped, 1 pre-existing failure inExecutionErrorDetailsTestCase.test_ai_log_skips_normal_content_after_an_error. The same failure reproduces on unmodified base8424dcbf(109 passed, 3 skipped, 1 failed). MiniMax API tests were skipped without credentials.The Hive regression tests execute the production definitions with a mocked cursor, following the existing connection-pool test pattern. Live Hive integration has not been tested.