job_runs: add lifecycle.triggers.on_file_change by radakam · Pull Request #6309 · databricks/cli

radakam · 2026-08-18T12:30:32Z

Changes

Add lifecycle.triggers.on_file_change for resources.job_runs (direct engine).

resources:
  job_runs:
    migrate:
      job_id: ${resources.jobs.migrate.id}
      lifecycle:
        triggers:
          - on_file_change: migrations/*.sql

Resolve the path or glob under the sync root and fingerprint each match by content hash, so an mtime-only touch does not re-fire.
Re-fire when a hash changes or when matches appear or disappear. Removing the trigger leaves the existing run alone: the plan reports it unchanged and no API call is made.
Skip whatever sync skips (.gitignore and sync.exclude).
Store the fingerprint as a flat lifecycle.triggers.on_file_change path-to-hash map, nested by value so a plan diff lands on the file.
Reject patterns outside the sync root, empty patterns, matches that are only directories or sync-excluded files, and combining a trigger with prevent_destroy. Warn when a pattern matches nothing, and when ** is used, since filepath.Glob treats it as a single *.

Why

Users need to re-run a job when local inputs change (SQL migrations, notebooks, configs) without re-firing on every deploy. Content hashes keep the trigger stable under sync noise while still catching real edits.

A pattern that resolves to nothing would silently disarm the trigger, so the cases that cannot re-arm later (directory-only and sync-excluded matches) are errors, while a pattern that matches no file yet is only a warning.

Tests

Unit: content hashing and pattern whitespace trimming in resolve_job_run_file_triggers; trigger validation for an empty pattern, both keys set, neither key set, on_bundle_deploy: false, and the prevent_destroy conflict.
Acceptance:
- on_file_change — first deploy fires; unchanged redeploy and touch plan nothing; an edit recreates (including READPLAN); deleting the file warns and re-fires; restoring it re-fires again; removing the trigger is unchanged with no run-now.
- on_file_change_glob — migrations/* plus mysubdir/*.txt; matches appearing and disappearing re-fire; files below the glob and gitignored files do not; ** warns and matches the same files as *; READPLAN.
- on_file_change_errors — pattern outside the sync root, a glob matching only directories, one matching a directory plus a sync-excluded file, one matching only sync-excluded files, and the no-match warning.
- destroy_unfinished_run — a run left going by an interrupted deploy is cancelled before it is deleted.
job_run_test.go shrinks where an acceptance test now asserts the same behavior, so coverage lives where the user-visible output is checked.

eng-dev-ecosystem-bot · 2026-08-18T13:54:48Z

Integration test report

Commit: 17857f9

Run: 32753813011

	Env	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
🔄	aws linux	3		4	274	1173	14:18
💚	aws windows		1	4	278	1171	7:09
💚	azure linux		1	4	275	1173	12:37
💚	azure windows		1	4	277	1171	6:36
💚	gcp linux		1	4	276	1173	10:55
💚	gcp windows		1	4	278	1171	7:06

7 interesting tests: 4 SKIP, 3 flaky

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
🔄	TestAccept	🔄f	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/bundle/invariant/no_drift	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🔄	TestAccept/bundle/resources/job_runs/failed_run	🔄f	✅p	✅p	✅p	✅p	✅p
🔄	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct	🔄f	✅p	✅p	✅p	✅p	✅p
🙈	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S

Top 15 slowest tests (at least 2 minutes):

duration	env	testname
3:57	aws windows	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
3:31	azure linux	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
3:17	azure windows	TestAccept
3:07	gcp windows	TestAccept
3:07	aws windows	TestAccept
3:07	azure windows	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
2:47	gcp windows	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
2:33	aws linux	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
2:27	gcp linux	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
2:24	azure linux	TestFilerReadDir/workspace_files
2:11	gcp linux	TestImportDirWithOverwriteFlag
2:05	azure windows	TestWorkspaceFilesExtensionsNotebooksAreNotReadAsFiles
2:05	azure windows	TestLockUnlockWithAllowsLockFileNotExist
2:03	aws linux	TestFilerWorkspaceFilesExtensionsReadDir
2:03	azure linux	TestImportDirWithOverwriteFlag

github-actions · 2026-08-19T08:05:38Z

Approval status: pending

`/acceptance/bundle/` - needs approval

58 files changed
Suggested: @janniklasrose
Also eligible: @denik, @pietern, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

`/bundle/` - needs approval

16 files changed
Suggested: @janniklasrose
Also eligible: @denik, @pietern, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

General files (require maintainer)

Files: .nextchanges/bundles/job-runs-on-file-change.md
Based on git history:

@janniklasrose -- recent work in bundle/config/mutator/, bundle/schema/, bundle/internal/schema/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.

See OWNERS for ownership rules.}

denik · 2026-08-20T12:56:28Z

+
+	t.Run("unset", func(t *testing.T) {
+		state := (&ResourceJobRun{}).PrepareState(&resources.JobRun{})
+		assert.Nil(t, state.Lifecycle)


not sure this adds coverage on top of acc tests?

could you review other tests in job_run_test.go as well; if they are not covered by acc tests, should we add acc test instead?

Good point, dropped duplicates and added destroy_unfinished_run acceptance test to convert a few more, I think it makes sense to keep the remaining tests here. What do you think?

Re-fire a run when matched file contents change, using stable hash fingerprints so mtime-only updates (touch) do not recreate.

Satisfy the linter on PrepareState literals after adding OnFileChange, and refresh out.fields.txt so validate-generated stays green.

Reject patterns outside the sync root and directory-only matches so the trigger cannot fingerprint escaped paths or silently disarm.

Drop size/mtime fingerprints and the resolve fast path so content identity alone drives recreate, and touch no longer needs a planner exception.

The modernize linter rejects the manual m[k]=v copy loop, failing CI lint.

Honor .gitignore and sync.exclude when hashing matches. Cover mysubdir globs, READPLAN, and keep LF on Windows so restored files hash cleanly.

The glob test edits migrations/ignored.txt, but the test .gitignore excluded it from the commit so CI clones would miss the file.

Plan serialization is already covered by on_file_change; the glob cases only assert the hash map.

Cover deploying glob hashes from a saved plan, not only from a live re-plan.

Keep the gitignore fixture as dot_gitignore so it does not affect the repo while developing, drop unit tests that duplicate acceptance coverage, and allocate trigger state once when a trigger is armed.

The previous simplification omitted OnBundleDeploy from the struct literal, which the linter rejects.

Always persist the same lifecycle.triggers shape so trigger fingerprints compare at the same paths. Wrap on_file_change hashes in files so dropping the trigger updates instead of recreating.

Nest lifecycle by value so structdiff descends to the leaf that changed instead of reporting the whole subtree, and let RemapState supply the empty shape. GetRun never returns the fingerprints, so the remote type no longer advertises a lifecycle it cannot fill.

Drop the files wrapper so plan diffs land on lifecycle.triggers.on_file_change, matching the config path without an extra state-only level.

The trigger, PrepareState and OverrideChangeDesc cases are asserted end to end by on_bundle_deploy and on_file_change, so the unit copies only duplicated goldens. Keep the ones acceptance cannot reach without stubbing GetRun.

Flattening on_file_change to a map made an unarmed trigger drop out of the serialized state, so the three plans that print an unarmed job_run now report an empty triggers object.

The unit test staged an asynchronous cancellation by hand; the acceptance test shows the real request order a user's destroy produces. Also drops two unit tests that the on_file_change acceptance tests already cover.

Directory-only and sync-excluded matches silently disarmed the trigger. Treat them as errors, warn that ** is not recursive, and cover both in acc.

The goldens were captured against a one-line-taller databricks.yml, so CI failed after the source settled on line 24.

Clearing a trigger is now a skip, so the plan reports the run as unchanged and the deploy makes no API call.

denik · 2026-08-24T09:17:57Z

+      "seed.txt": ""
+    }
+  },
+  "lifecycle.triggers.on_file_change['seed.txt']": {


Non blocking but still a bit weird that have both "lifecycle.triggers.on_file_change['seed.txt']" and "lifecycle.triggers.on_file_change" here.

Patterns must still land under the sync root. Reject **, warn when a glob matches directories, and skip only when the trigger itself is cleared.

radakam marked this pull request as ready for review August 19, 2026 08:04

Base automatically changed from deco-job-runs-lifecycle-on-bundle-deploy to main August 19, 2026 09:41

radakam force-pushed the deco-job-runs-lifecycle-on-file-change branch from b8c20f6 to 8744e5e Compare August 19, 2026 09:47

radakam requested a review from denik August 19, 2026 11:06

radakam force-pushed the deco-job-runs-lifecycle-on-file-change branch from 8744e5e to 47994d8 Compare August 19, 2026 11:39

denik reviewed Aug 19, 2026

View reviewed changes

Comment thread acceptance/bundle/resources/job_runs/on_file_change/output.txt

radakam requested a review from denik August 20, 2026 08:12

denik reviewed Aug 20, 2026

View reviewed changes

Comment thread acceptance/bundle/resources/job_runs/on_file_change_glob/test.toml

radakam requested a review from denik August 20, 2026 09:27

denik reviewed Aug 20, 2026

View reviewed changes

radakam requested a review from denik August 20, 2026 14:10

denik reviewed Aug 21, 2026

View reviewed changes

Comment thread bundle/direct/dresources/job_run.go Outdated

denik reviewed Aug 21, 2026

View reviewed changes

Comment thread bundle/direct/dresources/job_run.go

radakam requested a review from denik August 24, 2026 08:50

radakam added 13 commits August 24, 2026 13:11

job_runs: add lifecycle.triggers.on_file_change

d2ca935

Re-fire a run when matched file contents change, using stable hash fingerprints so mtime-only updates (touch) do not recreate.

job_runs: fix exhaustruct and regenerate refschema for on_file_change

b4665d0

Satisfy the linter on PrepareState literals after adding OnFileChange, and refresh out.fields.txt so validate-generated stays green.

job_runs: tighten on_file_change path and directory checks

ecf44ef

Reject patterns outside the sync root and directory-only matches so the trigger cannot fingerprint escaped paths or silently disarm.

job_runs: store on_file_change as path-to-hash only

4a0a2b4

Drop size/mtime fingerprints and the resolve fast path so content identity alone drives recreate, and touch no longer needs a planner exception.

job_runs: use maps.Copy in on_file_change hash merge

25f9566

The modernize linter rejects the manual m[k]=v copy loop, failing CI lint.

job_runs: drop filepath.Glob jargon from on_file_change schema docs

16367e6

acc: refresh on_file_change goldens for resource-key progress lines

83bbb02

acc: cover on_file_change globs, errors, and appear/disappear

1b73b00

acc: expand on_file_change glob coverage and skip ignored files

9e7c0fd

Honor .gitignore and sync.exclude when hashing matches. Cover mysubdir globs, READPLAN, and keep LF on Windows so restored files hash cleanly.

acc: force-add gitignored on_file_change fixture

4bf37e7

The glob test edits migrations/ignored.txt, but the test .gitignore excluded it from the commit so CI clones would miss the file.

acc: drop READPLAN from the on_file_change glob test

4a05a4c

Plan serialization is already covered by on_file_change; the glob cases only assert the hash map.

acc: restore READPLAN on the on_file_change glob test

59d34ca

Cover deploying glob hashes from a saved plan, not only from a live re-plan.

job_runs: address on_file_change review comments

f344a7e

Keep the gitignore fixture as dot_gitignore so it does not affect the repo while developing, drop unit tests that duplicate acceptance coverage, and allocate trigger state once when a trigger is armed.

radakam added 10 commits August 24, 2026 13:11

job_runs: list every trigger field to satisfy exhaustruct

3850eb8

The previous simplification omitted OnBundleDeploy from the struct literal, which the linter rejects.

job_runs: keep empty nested lifecycle on state and remote

b27c4ff

Always persist the same lifecycle.triggers shape so trigger fingerprints compare at the same paths. Wrap on_file_change hashes in files so dropping the trigger updates instead of recreating.

job_runs: store on_file_change hashes on the trigger itself

1364bf9

Drop the files wrapper so plan diffs land on lifecycle.triggers.on_file_change, matching the config path without an extra state-only level.

job_runs: drop unit tests already covered by acceptance

bd2b531

The trigger, PrepareState and OverrideChangeDesc cases are asserted end to end by on_bundle_deploy and on_file_change, so the unit copies only duplicated goldens. Keep the ones acceptance cannot reach without stubbing GetRun.

acc: refresh job_runs goldens for the omitted empty trigger

ee9196b

Flattening on_file_change to a map made an unarmed trigger drop out of the serialized state, so the three plans that print an unarmed job_run now report an empty triggers object.

job_runs: cover cancel-before-delete with an acceptance test

9e25ff8

The unit test staged an asynchronous cancellation by hand; the acceptance test shows the real request order a user's destroy produces. Also drops two unit tests that the on_file_change acceptance tests already cover.

job_runs: error when on_file_change hashes nothing

c255530

Directory-only and sync-excluded matches silently disarmed the trigger. Treat them as errors, warn that ** is not recursive, and cover both in acc.

acc: fix on_file_change_errors diagnostic line numbers

79b269e

The goldens were captured against a one-line-taller databricks.yml, so CI failed after the source settled on line 24.

acc: removing on_file_change is unchanged, not an update

4e7b960

Clearing a trigger is now a skip, so the plan reports the run as unchanged and the deploy makes no API call.

radakam force-pushed the deco-job-runs-lifecycle-on-file-change branch from 0756969 to 4e7b960 Compare August 24, 2026 13:24

denik reviewed Aug 24, 2026

View reviewed changes

radakam mentioned this pull request Aug 24, 2026

Job runs drop unit tests covered by acc #6370

Open

job_runs: resolve on_file_change relative to the defining YAML file

17857f9

Patterns must still land under the sync root. Reject **, warn when a glob matches directories, and skip only when the trigger itself is cleared.

Uh oh!

Conversation

radakam commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

eng-dev-ecosystem-bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

github-actions Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approval status: pending

/acceptance/bundle/ - needs approval

/bundle/ - needs approval

General files (require maintainer)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

denik Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

radakam Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

denik Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

radakam Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

denik Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

radakam commented Aug 18, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Aug 18, 2026 •

edited

Loading

github-actions Bot commented Aug 19, 2026 •

edited

Loading

`/acceptance/bundle/` - needs approval

`/bundle/` - needs approval