Publish DBR 18 as an umbrella line (18.x), not per point-release [DECO-28292 bug 2] - #26
Draft
rugpanov wants to merge 5 commits into
Draft
Publish DBR 18 as an umbrella line (18.x), not per point-release [DECO-28292 bug 2]#26rugpanov wants to merge 5 commits into
rugpanov wants to merge 5 commits into
Conversation
…nor [DECO-28292 bug 2]
Clusters on the DBR 18 LTS line report spark_version as the bare major
(dbr/18.x-scala2.13), but generation keyed folders off each point-release
page's own minor (18.0.x / 18.1.x / 18.2.x) — pages a cluster never
requests — so uv got a 404 (~25% of E_ENV_UNSUPPORTED "target valid but
no environment published" failures). The DBR 19 umbrella fallback only
triggered because 19 has no point-release pages yet.
DBR changed its versioning at 18: the index links a bare-major umbrella
slug ('18'/'18ml') whose point releases live at their own pages. Treat
major >= 18 as umbrella lines:
- _umbrella_major(index_slug) returns the bare major for these lines
(None for the pre-18 scheme, which keeps reading the minor from the
page title).
- sync_dbr / sync_dbr_ml key the folder by that bare major
('18.x-...'), while packages, Scala/Python and the databricks-connect
pin still come from the page.
- dbr_point_releases returns only the LATEST live (non-EoS) point
release for an umbrella line — the page a bare-major spark_version
resolves to (18.2 today; 18.0 is EoS) — so exactly one folder is
generated, pinning databricks-connect~=18.2.0 rather than the EoS
18.0 line the umbrella page's title would default to.
Regenerates python/dbr/18.x-{scala2.13,cpu-ml-scala2.13,gpu-ml-scala2.13}
(content identical to the latest maintenance 18.2) and removes the stale
18.0.x / 18.1.x / 18.2.x folders. DBR 19 behaviour is unchanged.
Co-authored-by: Isaac <no-reply@databricks.com>
…e is indeterminate dbr_point_releases now generates the single '<major>.x' folder from the latest live point release, so a transient fetch error on a release newer than the newest one confirmed live would let live[-1:] pick an older release — silently downgrading that folder's packages and databricks-connect pin (and the next --check would open a spurious downgrade PR). The old per-minor folders didn't have this: each minor owned its own folder. Track the minors that hit a transient error and, if any sits above the latest confirmed-live minor, return [] (skip the line this run) instead of downgrading; a clean run picks up the true latest. A transient below the confirmed latest can't change which release is newest, so it's ignored. Mirrors the repo's existing rule that a flaky probe is never mistaken for end-of-list. Also tightens the docstring: the function now returns a single-element list (or empty), not a variable-length one. Co-authored-by: Isaac <no-reply@databricks.com>
…ction Two robustness fixes to dbr_point_releases surfaced in review: - EoS point releases are no longer counted as end-of-list "misses". live_page now distinguishes 'eos' (retired but present — skip and keep probing) from 'absent' (a genuine 404, which counts). Before this, once two consecutive point releases went EoS (e.g. a future 18.0 + 18.1), the probe stopped before the live latest (18.2) and the umbrella folder was generated from the wrong (umbrella/.0) metadata. - When no release is confirmed live but a probe hit a transient error, we now skip the line instead of falling back to the umbrella page — we can't be sure the line has no point release, and guessing could generate from the wrong page. A line with genuinely no point releases (all clean 404s, e.g. DBR 19) still falls back to the umbrella page as before. Also makes `major` a required arg of _sync_dbr_ml_page (its only caller always passes it) and aligns the _umbrella_major regex comment with the repo style. Co-authored-by: Isaac <no-reply@databricks.com>
…contract Review follow-ups: - dbr_point_releases now falls back to the umbrella page only when the line has NO point-release pages at all (all genuine 404s, e.g. DBR 19 today). If point-release pages exist but every one is EoS, the line is retired: return [] rather than regenerate it from the umbrella page, which may not yet carry the EoS marker. Tracks `saw_page` (any live-or-EoS probe) to tell the two apart. - README: document that DBR 18+ umbrella lines are keyed by bare major and generated from the latest live point release, while pre-18 lines keep the minor from their page title. - test_sync: add the fully-EoS case and a sole-live-release guard. Co-authored-by: Isaac <no-reply@databricks.com>
Review follow-ups (no behaviour change): - dbr_point_releases now prints a diagnostic when it skips an umbrella line because a probe was indeterminate (matching the repo's `! dbr [...]` logging idiom), so operators can see why a line didn't sync in a flaky run. - Add the missing one-line docstring on the _slug_of test helper. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Bug 2 of DECO-28292. VPEX
E_ENV_UNSUPPORTEDtelemetry shows ~25% of "target valid but no environment published" failures are clusters on the DBR 18 LTS line: they reportspark_versionas the bare major (dbr/18.x-scala2.13), but generation keyed folders off each point-release page's own minor (18.0.x/18.1.x/18.2.x) — pages a cluster never requests — souvgot a 404. The DBR 19 umbrella fallback only worked because 19 has no point-release pages yet.Change
DBR changed its versioning at 18: the release-notes index links a bare-major umbrella slug (
18/18ml) whose point releases live at their own pages. This treats major ≥ 18 as umbrella lines:_umbrella_major(index_slug)— returns the bare major for these lines;Nonefor the pre-18 scheme (which keeps reading the minor from the page title, unchanged).sync_dbr/sync_dbr_ml— key the folder by that bare major (18.x-…), while packages, Scala/Python and thedatabricks-connectpin still come from the page.dbr_point_releases— returns only the latest live (non-EoS) point release for an umbrella line — the page a bare-majorspark_versionresolves to (18.2today;18.0is EoS) — so exactly one folder is generated, pinningdatabricks-connect~=18.2.0rather than the EoS18.0line the umbrella page's title would default to.Regenerates
python/dbr/18.x-{scala2.13,cpu-ml-scala2.13,gpu-ml-scala2.13}(content identical to the latest maintenance18.2) and removes the stale18.0.x/18.1.x/18.2.xfolders (9 folders). DBR 19 behaviour is unchanged.Tests
test_sync.py:_umbrella_major;dbr_point_releases(latest-only for umbrella18/18ml, umbrella fallback for19, pre-18 unchanged); andsync_dbr/sync_dbr_mlintegration asserting the folder is keyed18.x-…withdatabricks-connect~=18.2.0(fails on the old code, which produced18.2.x-…). All 33 tests pass.Acceptance criteria
dbr/18.x-scala2.13(+ ML variants) published; stale18.N.xfolders removed.uv— new18.xconstraints are byte-identical to the existing18.2.xfolders (samedatabricks-connect~=18.2.0), so resolution behaviour is identical;uv lockcould not be run in the authoring environment (pypi.orgunreachable there) — relying on CI on the real network.Notes
sync.py; base branch isdeco-28292-dbr-scala-variantsand will auto-retarget tomainwhen Publish DBR Scala 2.13 image when a runtime ships two Scala variants #25 merges.This pull request and its description were written by Isaac.