Publish DBR Scala 2.13 image when a runtime ships two Scala variants - #25
Open
rugpanov wants to merge 3 commits into
Open
Publish DBR Scala 2.13 image when a runtime ships two Scala variants#25rugpanov wants to merge 3 commits into
rugpanov wants to merge 3 commits into
Conversation
*Why*
dbr_meta read the Scala version with a single re.search, taking only the first
version a page lists. A runtime mid-migration publishes two images from one page
("Scala: 2.12.15 or 2.13.10"), but only the 2.12 folder was generated -- so a
cluster on the Scala 2.13 image (spark_version 16.4.x-scala2.13) resolves to no
published environment and 404s. This is the largest E_ENV_UNSUPPORTED source in
the VPEX telemetry (~69%: 16.4.x-scala2.13 + its cpu-ml variant), across many
distinct workspaces. DBR 16.4 LTS is the only dual-image line today; the fix is
version-agnostic so future ones are covered automatically.
*What*
- sync.py: add dbr_scalas(), which returns every Scala MAJOR.MINOR a page's System
environment lists (one entry for a single-image page, both for a dual-image one),
de-duped and in page order, with a fallback to the prior single-match. dbr_meta
returns that list; sync_dbr and the ML path write one folder per Scala version off
the shared "Installed Python libraries" table (only the Java/Scala tables split per
image, which this repo doesn't consume).
- envgen.py: extend DROP_BY_ENV with the three 16.4.x...scala2.13 keys. The pandas
1.5.x-on-cp312 problem (issue #18) is Scala-independent, so the 2.13 image needs the
same drop as the 2.12 image -- without it the new folders re-break #18.
- Regenerated artifacts: dbr/16.4.x-scala2.13, 16.4.x-cpu-ml-scala2.13,
16.4.x-gpu-ml-scala2.13. Byte-identical pin sets to their 2.12 siblings; all three
pass full `uv sync`.
- Tests: new test_sync.py (dbr_scalas/dbr_meta on dual/single/missing/malformed
input); extend the envgen pandas-drop test to the scala2.13 envs.
Co-authored-by: Isaac <no-reply@databricks.com>
Review hardening: dbr_scalas captured the raw HTML between 'Scala</strong>:' and </li>, so a version-like number inside a tag (e.g. an href to a Spark-version doc page) could be read as a Scala version and spawn a nonexistent environment. Strip inline tags first -- the same treatment table_pkgs already applies -- so only the visible version tokens count. No generated artifact changes (real pages carry no such tag digits today); adds a regression test. Co-authored-by: Isaac <no-reply@databricks.com>
…variant Review follow-up. The previous parser read every version-like token in the Scala field, so a trailing visible annotation (e.g. "2.12.15 or 2.13.10 (Apache Spark 3.5)") would add a bogus "3.5" and generate a nonexistent scala3.5 environment. Match only the leading "VER (or VER)*" enumeration -- the Scala version, or the two joined by "or" for a dual-image release -- and stop at any following text. Tag stripping is kept so the enumeration survives the "<strong>or</strong>" markup. Adds a regression test for the trailing-annotation case; no generated artifact changes. Also update README: the DBR sync description now notes that a dual-image release produces one environment per Scala version off the page's single Python table. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
dbr_metaread the Scala version with a singlere.search, taking only the first version a runtime page lists. A runtime mid-migration publishes two images from one page (Scala: 2.12.15 or 2.13.10), but only the 2.12 folder was generated — so a cluster on the Scala 2.13 image (spark_version = 16.4.x-scala2.13) resolves to no published environment and 404s.This is the largest
E_ENV_UNSUPPORTEDsource in the VPEX telemetry (~69%:16.4.x-scala2.13+ itscpu-mlvariant), spread across many distinct workspaces. DBR 16.4 LTS is the only dual-image line today; the fix is version-agnostic so any future dual-image runtime is covered automatically.What
sync.py— adddbr_scalas(), returning every ScalaMAJOR.MINORa page's System environment lists (one entry for a single-image page, both for a dual-image one), de-duped and in page order, with a fallback to the prior single-match if the field isn't</li>-delimited.dbr_metareturns that list;sync_dbrand the ML path write one folder per Scala version off the sharedinstalled-python-librariestable (only the Java/Scala tables split per image, which this repo doesn't consume).envgen.py— extendDROP_BY_ENVwith the three16.4.x…scala2.13keys. Thepandas 1.5.x-on-cp312 problem (dbr/16.4.x, serverless-v3: pandas 1.5.3 can't be installed on Python 3.12 (no cp312 wheel) #18) is Scala-independent, so the 2.13 image needs the same drop as the 2.12 image — without it the new folders would re-break dbr/16.4.x, serverless-v3: pandas 1.5.3 can't be installed on Python 3.12 (no cp312 wheel) #18.dbr/16.4.x-scala2.13,dbr/16.4.x-cpu-ml-scala2.13,dbr/16.4.x-gpu-ml-scala2.13. Pin sets are byte-identical to their 2.12 siblings (only the name strings differ).test_sync.py(dbr_scalas/dbr_metaon dual/single/missing/malformed input); extend theenvgenpandas-drop test to the scala2.13 envs.Verification
python -m unittest test_sync test_envgen→ 22 passing (CI'stest.ymlauto-discoverstest_sync.py).python .github/scripts/sync.py --check→ "no changes — repo is in sync with published docs", exit 0.uv sync(build + install) PASSES for all three new envs via the Databricks PyPI proxy.This pull request and its description were written by Isaac.