Skip to content

Add job_runs lifecycle.triggers.on_value_change - #6341

Open
radakam wants to merge 24 commits into
deco-job-runs-lifecycle-on-file-changefrom
deco-job-runs-lifecycle-on-value-change
Open

Add job_runs lifecycle.triggers.on_value_change#6341
radakam wants to merge 24 commits into
deco-job-runs-lifecycle-on-file-changefrom
deco-job-runs-lifecycle-on-value-change

Conversation

@radakam

@radakam radakam commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Changes

Add lifecycle.triggers.on_value_change for resources.job_runs (direct engine). The run is recreated when the resolved expression changes. ${resources.*} refs become deploy-graph dependencies. Multiple triggers are separate list entries (one key each). Removing the trigger is unchanged: it does not recreate or start a new run (same as on_file_change / on_bundle_deploy).

Why

job_runs already support on_bundle_deploy and on_file_change. Watching a resolved value (variable or resource id) is the remaining trigger from the job_runs design, so a run can re-fire when an upstream resource or interpolated value changes.

Tests

  • Unit: validate_job_run_triggers (empty/whitespace/two-keys/prevent_destroy). Wait/GetRun stubs stay in job_run_test.go; PrepareInputConfig / DropJobRunValueChangeConfigRefs are covered by acceptance.
  • Acceptance on_value_change: two expressions; changing one var recreates (other fingerprint stays); removing both triggers is unchanged with no run-now.
  • Acceptance on_value_change_ref: concatenated ${resources.jobs.other.id}-${resources.jobs.extra.id}; depends_on includes both jobs; persisted state is [OTHER_ID]-[EXTRA_ID]; READPLAN="" and READPLAN=1.

radakam added 16 commits August 19, 2026 11:36
Re-fire a run when matched file contents change, using stable
hash fingerprints so mtime-only updates (touch) do not recreate.
Satisfy the linter on PrepareState literals after adding OnFileChange,
and refresh out.fields.txt so validate-generated stays green.
Reject patterns outside the sync root and directory-only matches so
the trigger cannot fingerprint escaped paths or silently disarm.
Drop size/mtime fingerprints and the resolve fast path so content
identity alone drives recreate, and touch no longer needs a planner exception.
The modernize linter rejects the manual m[k]=v copy loop, failing CI lint.
Honor .gitignore and sync.exclude when hashing matches. Cover mysubdir
globs, READPLAN, and keep LF on Windows so restored files hash cleanly.
The glob test edits migrations/ignored.txt, but the test .gitignore
excluded it from the commit so CI clones would miss the file.
Plan serialization is already covered by on_file_change; the glob cases only assert the hash map.
Cover deploying glob hashes from a saved plan, not only from a live re-plan.
Keep the gitignore fixture as dot_gitignore so it does not affect the repo while developing, drop unit tests that duplicate acceptance coverage, and allocate trigger state once when a trigger is armed.
The previous simplification omitted OnBundleDeploy from the struct literal, which the linter rejects.
Always persist the same lifecycle.triggers shape so trigger fingerprints
compare at the same paths. Wrap on_file_change hashes in files so dropping
the trigger updates instead of recreating.
Nest lifecycle by value so structdiff descends to the leaf that changed
instead of reporting the whole subtree, and let RemapState supply the empty
shape. GetRun never returns the fingerprints, so the remote type no longer
advertises a lifecycle it cannot fill.
@radakam
radakam marked this pull request as ready for review August 21, 2026 15:59
@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Approval status: pending

/acceptance/bundle/ - needs approval

10 files changed
Suggested: @denik
Also eligible: @janniklasrose, @pietern, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

/bundle/ - needs approval

8 files changed
Suggested: @denik
Also eligible: @janniklasrose, @pietern, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

General files (require maintainer)

Files: .nextchanges/bundles/job-runs-on-value-change.md
Based on git history:

  • @denik -- recent work in bundle/direct/, .nextchanges/bundles/, bundle/internal/schema/

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.
See OWNERS for ownership rules.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 3865b6b

Run: 32739403320

Env 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 1 4 276 1172 10:02
💚​ aws windows 1 4 278 1170 6:14
💚​ azure linux 1 4 275 1172 9:56
💚​ azure windows 1 4 277 1170 7:46
💚​ gcp linux 1 4 276 1172 12:02
💚​ gcp windows 1 4 278 1170 7:28
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
Top 12 slowest tests (at least 2 minutes):
duration env testname
4:16 azure windows TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
4:10 gcp windows TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
3:27 aws windows TestAccept
3:25 azure windows TestAccept
3:17 aws linux TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
3:07 gcp windows TestAccept
2:59 gcp linux TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
2:47 azure linux TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
2:43 gcp linux TestFilerReadDir/workspace_files_extensions
2:41 aws windows TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
2:17 gcp windows TestFilerWorkspaceFilesExtensionsStat
2:15 azure linux TestFilerWorkspaceFilesExtensionsReadDir

Drop the files wrapper so plan diffs land on lifecycle.triggers.on_file_change, matching the config path without an extra state-only level.
The trigger, PrepareState and OverrideChangeDesc cases are asserted end to end by on_bundle_deploy and on_file_change, so the unit copies only duplicated goldens. Keep the ones acceptance cannot reach without stubbing GetRun.
Flattening on_file_change to a map made an unarmed trigger drop out of the serialized state, so the three plans that print an unarmed job_run now report an empty triggers object.
The unit test staged an asynchronous cancellation by hand; the acceptance
test shows the real request order a user's destroy produces. Also drops two
unit tests that the on_file_change acceptance tests already cover.
Re-fire a run when a resolved expression changes, including resource refs that become deploy-graph edges. Removing the trigger is a state-only update.
Include the on_value_change input and state paths required by generated-file validation.
@radakam
radakam force-pushed the deco-job-runs-lifecycle-on-value-change branch from 0db1f4b to e431088 Compare August 24, 2026 09:36
@radakam
radakam force-pushed the deco-job-runs-lifecycle-on-file-change branch from 0756969 to 4e7b960 Compare August 24, 2026 13:24
…s-lifecycle-on-value-change

Keep on_value_change on top of the latest on_file_change base, including skip when a trigger is removed.
on_value_change_ref already asserts depends_on and persisted resolved ids.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants