acceptance: skip default-python classic-compute tests on UC envs#5153
Draft
shreyas-goenka wants to merge 5 commits intomainfrom
Draft
acceptance: skip default-python classic-compute tests on UC envs#5153shreyas-goenka wants to merge 5 commits intomainfrom
shreyas-goenka wants to merge 5 commits intomainfrom
Conversation
The default-python template's pipeline resource defaults to a hive_metastore
catalog when no UC catalog is detected on the workspace (see
libs/template/templates/default/template/{{.project_name}}/resources/{{.project_name}}_etl.pipeline.yml.tmpl).
Several of the test workspaces (gcp-prod-is, aws-prod-is) have HMS
disabled, so deploys intermittently fail with:
Error: cannot create resources.pipelines.X_etl: Hive Metastore is not
enabled for this workspace; please publish to Unity Catalog. (400
INVALID_PARAMETER_VALUE)
Set RequiresUnityCatalog = true on integration_classic and
combinations/classic so they only run on UC-enabled workspaces, where
default_catalog resolves to a real UC catalog and the pipeline gets a
proper "catalog:" field. The serverless variant of combinations is
unaffected because the template hardcodes "catalog: main" for serverless
pipelines without UC.
Co-authored-by: Isaac
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Co-authored-by: Isaac
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Co-authored-by: Isaac
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
…mments Co-authored-by: Isaac
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
…sic tests Inverts the gate: keep coverage on non-UC envs and skip on UC envs where default_catalog returns "hive_metastore" but no such catalog exists. Co-authored-by: Isaac
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
default-pythonclassic-compute template tests fail on UC test workspaces because the pipeline template (source) omitscatalog:whendefault_catalogresolves to""or"hive_metastore"— which it does on*-prod-ucws-is, where the metastore-assignment carries the legacy literaldefault_catalog_name = "hive_metastore". Without an explicit catalog, DLT falls back tohive_metastore, which fails on UC-only workspaces withHive Metastore is not enabled for this workspace.The HMS-disable on UC workspaces might not have been rolled out to all workspaces yet (same dynamic as #5106's note that
RESOURCE_ALREADY_EXISTShas not been rolled out to all workspaces yet). The same workspace can pass one nightly and fail the next as the rollout progresses, so a defensive skip is appropriate even when current samples happen to be green.This PR adds a
SkipOnUnityCatalogtest config field (inverse ofRequiresUnityCatalog) and sets it onintegration_classicandcombinations/classic. Coverage is preserved on non-UC envs. Drop the skip once UC reconcilesdefault_catalog_name(filed with `#eng-unity-catalog`).The
serverlessvariant ofcombinationsis unaffected — the template hardcodescatalog: mainthere.Test plan
This pull request was AI-assisted by Isaac.