job_runs: add lifecycle.triggers.on_file_change #6309
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| direct: `resources.job_runs` can set `lifecycle.triggers.on_file_change` to a path or glob under the sync root to re-fire the run when matched file contents change (content hash), or when matches appear or disappear. Removing the trigger does not recreate the existing run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| bundle: | ||
| name: job-runs-on-file-change | ||
|
|
||
| resources: | ||
| jobs: | ||
| my_job: | ||
| name: my-job | ||
| tasks: | ||
| - task_key: main | ||
| notebook_task: | ||
| notebook_path: /Workspace/test | ||
|
|
||
| job_runs: | ||
| my_run: | ||
| job_id: ${resources.jobs.my_job.id} | ||
| lifecycle: | ||
| triggers: | ||
| - on_file_change: seed.txt | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,207 @@ | ||
|
|
||
| === first deploy triggers a run | ||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/job-runs-on-file-change/default/files... | ||
| Output from job_runs.my_run: id=[MY_RUN_ID]: Run URL: [DATABRICKS_URL]/jobs/[MY_JOB_ID]/runs/[MY_RUN_ID]?o=[NUMID] | ||
| Output from job_runs.my_run: id=[MY_RUN_ID]: SUCCESS | ||
| Created job_runs.my_run | ||
| Created jobs.my_job | ||
| Files: 6 uploaded, 0 deleted | ||
| Resources: 2 created, 0 changed, 0 deleted, 0 unchanged | ||
|
|
||
| >>> read_id.py my_job | ||
| [MY_JOB_ID] | ||
|
|
||
| >>> print_requests.py //jobs/run-now | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.2/jobs/run-now", | ||
| "body": { | ||
| "idempotency_token": "[UUID]", | ||
| "job_id": [MY_JOB_ID] | ||
| } | ||
| } | ||
|
|
||
| === redeploy with unchanged file plans nothing | ||
| >>> [CLI] bundle plan | ||
| Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/job-runs-on-file-change/default/files... | ||
| Files: 2 uploaded, 0 deleted | ||
| Resources: 0 created, 0 changed, 0 deleted, 2 unchanged | ||
|
|
||
| >>> print_requests.py //jobs/run-now | ||
|
|
||
| === touch without content change plans nothing | ||
| >>> [CLI] bundle plan | ||
| Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/job-runs-on-file-change/default/files... | ||
| Files: 3 uploaded, 0 deleted | ||
| Resources: 0 created, 0 changed, 0 deleted, 2 unchanged | ||
|
|
||
| >>> print_requests.py //jobs/run-now | ||
|
|
||
| === editing the file re-fires | ||
| >>> [CLI] bundle plan | ||
| recreate job_runs.my_run | ||
|
radakam marked this conversation as resolved.
|
||
|
|
||
| Plan: 1 to add, 0 to change, 1 to delete, 1 unchanged | ||
|
|
||
| >>> jq .plan["resources.job_runs.my_run"].changes tmp.plan.json | ||
| { | ||
| "lifecycle.triggers.on_file_change": { | ||
| "action": "recreate", | ||
| "reason": "immutable", | ||
| "old": { | ||
|
radakam marked this conversation as resolved.
|
||
| "seed.txt": "[FILE_HASH][0]" | ||
| }, | ||
| "new": { | ||
| "seed.txt": "[FILE_HASH][1]" | ||
| } | ||
| }, | ||
| "lifecycle.triggers.on_file_change['seed.txt']": { | ||
| "action": "recreate", | ||
| "reason": "immutable", | ||
| "old": "[FILE_HASH][0]", | ||
| "new": "[FILE_HASH][1]" | ||
| }, | ||
| "result_state": { | ||
| "action": "skip", | ||
| "reason": "remote_already_set", | ||
| "new": "SUCCESS", | ||
| "remote": "SUCCESS" | ||
| } | ||
| } | ||
|
|
||
| === bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/job-runs-on-file-change/default/files... | ||
| Output from job_runs.my_run: id=[MY_RUN_ID_2]: Run URL: [DATABRICKS_URL]/jobs/[MY_JOB_ID]/runs/[MY_RUN_ID_2]?o=[NUMID] | ||
| Output from job_runs.my_run: id=[MY_RUN_ID_2]: SUCCESS | ||
| Recreated job_runs.my_run | ||
| Files: 4 uploaded, 0 deleted | ||
| Resources: 1 created, 0 changed, 1 deleted, 1 unchanged | ||
|
|
||
| >>> print_requests.py --keep //jobs/runs/delete | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.2/jobs/runs/delete", | ||
| "body": { | ||
| "run_id": [MY_RUN_ID] | ||
| } | ||
| } | ||
|
|
||
| >>> print_requests.py //jobs/run-now | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.2/jobs/run-now", | ||
| "body": { | ||
| "idempotency_token": "[UUID]", | ||
| "job_id": [MY_JOB_ID] | ||
| } | ||
| } | ||
|
|
||
| === deleting the file warns and re-fires | ||
| >>> [CLI] bundle plan | ||
| Warning: lifecycle.triggers.on_file_change: no files match "seed.txt" | ||
| in databricks.yml:18:29 | ||
|
|
||
| recreate job_runs.my_run | ||
|
|
||
| Plan: 1 to add, 0 to change, 1 to delete, 1 unchanged | ||
|
|
||
| >>> changes | ||
| Warning: lifecycle.triggers.on_file_change: no files match "seed.txt" | ||
| in databricks.yml:18:29 | ||
|
|
||
| { | ||
| "lifecycle.triggers.on_file_change": { | ||
| "action": "recreate", | ||
| "reason": "immutable", | ||
| "old": { | ||
| "seed.txt": "[FILE_HASH][1]" | ||
| }, | ||
| "new": { | ||
| "seed.txt": "" | ||
| } | ||
| }, | ||
| "lifecycle.triggers.on_file_change['seed.txt']": { | ||
|
Contributor
There was a problem hiding this comment. Non blocking but still a bit weird that have both |
||
| "action": "recreate", | ||
| "reason": "immutable", | ||
| "old": "[FILE_HASH][1]", | ||
| "new": "" | ||
| }, | ||
| "result_state": { | ||
| "action": "skip", | ||
| "reason": "remote_already_set", | ||
| "new": "SUCCESS", | ||
| "remote": "SUCCESS" | ||
| } | ||
| } | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Warning: lifecycle.triggers.on_file_change: no files match "seed.txt" | ||
| in databricks.yml:18:29 | ||
|
|
||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/job-runs-on-file-change/default/files... | ||
| Output from job_runs.my_run: id=[MY_RUN_ID_3]: Run URL: [DATABRICKS_URL]/jobs/[MY_JOB_ID]/runs/[MY_RUN_ID_3]?o=[NUMID] | ||
| Output from job_runs.my_run: id=[MY_RUN_ID_3]: SUCCESS | ||
| Recreated job_runs.my_run | ||
| Files: 3 uploaded, 1 deleted | ||
| Resources: 1 created, 0 changed, 1 deleted, 1 unchanged | ||
|
|
||
| >>> print_requests.py //jobs/run-now | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.2/jobs/run-now", | ||
| "body": { | ||
| "idempotency_token": "[UUID]", | ||
| "job_id": [MY_JOB_ID] | ||
| } | ||
| } | ||
|
|
||
| === restoring the file re-fires again | ||
| >>> [CLI] bundle plan | ||
| recreate job_runs.my_run | ||
|
|
||
| Plan: 1 to add, 0 to change, 1 to delete, 1 unchanged | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/job-runs-on-file-change/default/files... | ||
| Output from job_runs.my_run: id=[MY_RUN_ID_4]: Run URL: [DATABRICKS_URL]/jobs/[MY_JOB_ID]/runs/[MY_RUN_ID_4]?o=[NUMID] | ||
| Output from job_runs.my_run: id=[MY_RUN_ID_4]: SUCCESS | ||
| Recreated job_runs.my_run | ||
| Files: 3 uploaded, 1 deleted | ||
| Resources: 1 created, 0 changed, 1 deleted, 1 unchanged | ||
|
|
||
| >>> print_requests.py //jobs/run-now | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.2/jobs/run-now", | ||
| "body": { | ||
| "idempotency_token": "[UUID]", | ||
| "job_id": [MY_JOB_ID] | ||
| } | ||
| } | ||
|
|
||
| === removing on_file_change is unchanged and does not run | ||
| >>> [CLI] bundle plan | ||
| Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/job-runs-on-file-change/default/files... | ||
| Files: 3 uploaded, 0 deleted | ||
| Resources: 0 created, 0 changed, 0 deleted, 2 unchanged | ||
|
|
||
| >>> print_requests.py //jobs/run-now | ||
|
|
||
| >>> [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete resources.job_runs.my_run | ||
| delete resources.jobs.my_job | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/job-runs-on-file-change/default | ||
|
|
||
| Destroy: 2 deleted | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| cleanup() { | ||
| trace $CLI bundle destroy --auto-approve | ||
| rm -f out.requests.txt | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| changes() { | ||
| $CLI bundle plan -o json | jq '.plan["resources.job_runs.my_run"].changes' | ||
| } | ||
|
|
||
| title "first deploy triggers a run" | ||
| trace $CLI bundle deploy | ||
| trace read_id.py my_job | ||
| # Name the first run so the recreated one becomes [MY_RUN_ID_2]. | ||
| read_id.py my_run > /dev/null | ||
| trace print_requests.py //jobs/run-now | ||
|
|
||
| title "redeploy with unchanged file plans nothing" | ||
| trace $CLI bundle plan | ||
| trace $CLI bundle deploy | ||
| trace print_requests.py //jobs/run-now | ||
|
|
||
| title "touch without content change plans nothing" | ||
| python -c "import os; os.utime('seed.txt', None)" | ||
| trace $CLI bundle plan | ||
| trace $CLI bundle deploy | ||
| trace print_requests.py //jobs/run-now | ||
|
|
||
| title "editing the file re-fires" | ||
| update_file.py seed.txt "v1" "v2" | ||
| trace $CLI bundle plan | ||
| # Save the plan so the READPLAN=1 variant deploys the hashes computed here. | ||
| # The deploy is not traced: readplanarg makes the command line differ per variant. | ||
| $CLI bundle plan -o json > tmp.plan.json | ||
| trace jq '.plan["resources.job_runs.my_run"].changes' tmp.plan.json | ||
| title "bundle deploy\n" | ||
| $CLI bundle deploy $(readplanarg tmp.plan.json) | ||
| read_id.py my_run > /dev/null | ||
| trace print_requests.py --keep //jobs/runs/delete | ||
| trace print_requests.py //jobs/run-now | ||
|
radakam marked this conversation as resolved.
|
||
|
|
||
| title "deleting the file warns and re-fires" | ||
| mv seed.txt tmp.seed.txt | ||
| trace $CLI bundle plan | ||
| trace changes | ||
| trace $CLI bundle deploy | ||
| read_id.py my_run > /dev/null | ||
| trace print_requests.py //jobs/run-now | ||
|
|
||
| title "restoring the file re-fires again" | ||
| mv tmp.seed.txt seed.txt | ||
| trace $CLI bundle plan | ||
| trace $CLI bundle deploy | ||
| read_id.py my_run > /dev/null | ||
| trace print_requests.py //jobs/run-now | ||
|
|
||
| title "removing on_file_change is unchanged and does not run" | ||
| update_file.py databricks.yml " lifecycle: | ||
| triggers: | ||
| - on_file_change: seed.txt | ||
| " "" | ||
| trace $CLI bundle plan | ||
| trace $CLI bundle deploy | ||
| trace print_requests.py //jobs/run-now | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| v1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # Deploy both by re-planning and from a plan saved on disk, so the hashes | ||
| # computed during planning survive plan serialization. | ||
| EnvMatrix.READPLAN = ["", "1"] | ||
|
|
||
| Ignore = ["tmp.plan.json", "tmp.seed.txt"] | ||
|
|
||
| # Content hashes are stable but unreadable, and the parent's `\d{8,}` rule | ||
| # mangles any digit run inside them. Order=1 runs before it; Distinct keeps | ||
| # different hashes distinguishable. | ||
| [[Repls]] | ||
| Old = '[0-9a-f]{64}' | ||
| New = '[FILE_HASH]' | ||
| Order = 1 | ||
| Distinct = true |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.