Skip to content

air: bound submission config validation timeout and retry - #6994

Merged
caroline-db merged 3 commits into
mainfrom
air/preflight-validation-timeout
Oct 9, 2026
Merged

air: bound submission config validation timeout and retry#6994
caroline-db merged 3 commits into
mainfrom
air/preflight-validation-timeout

Conversation

@caroline-db

@caroline-db caroline-db commented Oct 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What did you change, and why?

  • Bound preflight validation for normal databricks air run submissions to one request with a five-second deadline after SDK credential setup. On main, this path uses the SDK's retry policy and can wait for the configured retry window, which defaults to five minutes, on retryable failures.
  • Warn and continue when validation is unavailable because the endpoint is disabled or missing, the service returns HTTP 408, 429, or 5xx, the request or token refresh reaches its deadline, transport fails, or the response cannot be read or decoded. Submission still receives authoritative downstream validation, so temporary preflight unavailability should not block it.
  • Continue blocking field-validation errors, HTTP 400/401/403 and other caller errors, definitive authentication failures, request setup failures, and user cancellation. These failures are actionable or explicitly initiated and should not be treated as service unavailability.
  • Preserve the validation request's SDK authentication, routing, user-agent, and trace headers, and add a release note for the bounded behavior.

How do you know it works?

  • Added focused coverage for the single-attempt limit, timeout, warning and error classification, OAuth refresh and rejection, response-body failures, malformed responses, and cancellation.
  • Passed go test ./cmd/air, focused race tests, go vet ./cmd/air, and GolangCI lint with Staticcheck disabled.
  • Verified authenticated workspace dry runs and a real A10G workload through successful completion and log streaming.

This PR was written with Codex.

@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Oct 8, 2026
@caroline-db caroline-db changed the title air: bound submission config validation to five seconds air: bound submission config validation timeout and retry Oct 8, 2026
@caroline-db
caroline-db force-pushed the air/preflight-validation-timeout branch from 8bf0bbd to 7d03020 Compare October 8, 2026 18:38
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 2bc3197

Run: 37992030096

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 276 16 4:20
✅​ aws-windows-latest-4core-16gb 278 14 4:21
✅​ azure linux-2core-8gb 275 16 4:18
✅​ azure-windows-latest-4core-16gb 277 14 3:35
✅​ gcp linux-2core-8gb 276 16 4:16
✅​ gcp-windows-latest-4core-16gb 278 14 3:37
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:09 aws-windows-latest-4core-16gb TestAccept
3:35 gcp-windows-latest-4core-16gb TestAccept
3:32 azure-windows-latest-4core-16gb TestAccept

@caroline-db
caroline-db marked this pull request as ready for review October 8, 2026 22:41
@caroline-db
caroline-db requested review from a team as code owners October 8, 2026 22:41
@caroline-db
caroline-db requested a review from rugpanov October 8, 2026 22:41
@rugpanov
rugpanov removed their request for review October 9, 2026 08:39
maggiewang-db
maggiewang-db reviewed Oct 9, 2026
View reviewed changes
Comment thread cmd/air/validateconfig.go
Comment thread cmd/air/validateconfig.go
Comment thread cmd/air/validateconfig.go
maggiewang-db
maggiewang-db reviewed Oct 9, 2026
View reviewed changes
Comment thread cmd/air/validateconfig.go
Comment thread cmd/air/validateconfig.go
@caroline-db
caroline-db added this pull request to the merge queue Oct 9, 2026
Merged via the queue into main with commit 146e947 Oct 9, 2026
26 checks passed
@caroline-db
caroline-db deleted the air/preflight-validation-timeout branch October 9, 2026 22:55
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 146e947

Run: 38001844142

Env ❌​FAIL 🔄​flaky ✅​pass 🙈​skip Time
🔄​ aws linux-2core-8gb 3 1425 1134 123:46
✅​ aws-windows-latest-4core-16gb 1353 1159 139:43
❌​ azure linux-2core-8gb 12 1265 1186 104:31
❌​ azure-windows-latest-4core-16gb 12 1190 1211 122:27
✅​ gcp linux-2core-8gb 1266 1190 114:52
🔄​ gcp-windows-latest-4core-16gb 3 1188 1215 126:29
16 interesting tests: 12 FAIL, 4 flaky
Test Name aws linux-2core-8gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp-windows-latest-4core-16gb
❌​ TestAccept 🔄​f ❌​F ❌​F 🔄​f
🔄​ TestAccept/bundle/apps/compute_size 🔄​f ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/apps/compute_size/DMS=true 🔄​f ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/deployment/bind/alert ✅​p ❌​F ❌​F ✅​p
❌​ TestAccept/bundle/deployment/bind/alert/DMS= ✅​p ❌​F ❌​F ✅​p
❌​ TestAccept/bundle/generate/alert ✅​p ❌​F ❌​F ✅​p
❌​ TestAccept/bundle/generate/alert/DMS= ✅​p ❌​F ❌​F ✅​p
❌​ TestAccept/bundle/generate/alert/DMS=true ✅​p ❌​F ❌​F ✅​p
🔄​ TestAccept/bundle/invariant/no_drift ✅​p ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/invariant/no_drift/DMS=/INPUT_CONFIG=secret.yml.tmpl/READPLAN= ✅​p ✅​p ✅​p 🔄​f
❌​ TestAccept/bundle/resources/alerts/basic ✅​p ❌​F ❌​F ✅​p
❌​ TestAccept/bundle/resources/alerts/basic/DMS= ✅​p ❌​F ❌​F ✅​p
❌​ TestAccept/bundle/resources/alerts/basic/DMS=true ✅​p ❌​F ❌​F ✅​p
❌​ TestAccept/bundle/resources/alerts/with_file ✅​p ❌​F ❌​F ✅​p
❌​ TestAccept/bundle/resources/alerts/with_file/DMS= ✅​p ❌​F ❌​F ✅​p
❌​ TestAccept/bundle/resources/alerts/with_file/DMS=true ✅​p ❌​F ❌​F ✅​p
Top 50 slowest tests (at least 2 minutes):
duration env testname
13:51 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
12:39 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
11:47 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
10:37 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
10:21 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
10:19 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
10:19 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
9:07 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
8:50 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
8:45 aws linux-2core-8gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
8:41 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
8:01 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
7:58 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:44 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:36 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:33 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:26 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:24 aws linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true
7:23 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:15 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:10 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
7:09 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:06 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:04 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:01 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
6:59 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:53 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
6:47 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:43 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:38 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:37 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:20 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:12 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:11 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:06 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:04 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:55 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:52 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:49 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:44 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:41 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:09 aws-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=
5:04 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=
5:00 azure-windows-latest-4core-16gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
4:59 aws linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
4:58 azure linux-2core-8gb TestAccept/bundle/deploy/spark-jar-task/DMS=
4:50 gcp-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
4:46 azure-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
4:45 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true
4:41 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maggiewang-db maggiewang-db maggiewang-db approved these changes

Assignees

No one assigned

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@caroline-db @eng-dev-ecosystem-bot @maggiewang-db