Repository navigation
Add VM image rollback script and presubmit failure decision step - #2986
Conversation
ted-xie
left a comment
ted-xie
left a comment
There was a problem hiding this comment.
you closed the PR before I could finish typing, but I had some high level questions here. but this looks great and I'm looking forward to the final version!
|
@ted-xie Ooops sorry, my agent decided it's ready for review on my behalf :D |
|
Updated and tested |
7e6b042 to
1388822
Compare
|
What happens if we identify some issue after promoting them to prod, can we still rollback with this script/process? |
Add buildkite/rollback_vm_image.py to deprecate the newest active READY image in each GCE image family in favor of the previous one and print the undo and create_instances.py commands. When validation presubmits fail in trigger_testing_presubmits.py, keep the error annotation with the infra-vs-test hint, upload pipelines/vm-image-rollback.yml with ROLLBACK_FAMILIES and ROLLBACK_GROUPS interpolated for the target org, and exit 0 so the pipeline pauses at a block step offering rollback or continue.
Expand the rollback plan comprehension into a plain loop and add short section comments in main() so the flow and fail-early checks are clear.
1388822 to
bc399d3
Compare
Yes, we can run this script manually. I added a description of how to use it at the top of the file. |
| ["buildkite-agent", "pipeline", "upload", "../pipelines/vm-image-rollback.yml"], | ||
| env={**os.environ, "ROLLBACK_FAMILIES": families, "ROLLBACK_GROUPS": groups}, | ||
| ) | ||
| return 0 |
There was a problem hiding this comment.
This returns 0 when the presubmits failed, so the step shows as passed in the build history. I understand this is needed for the "continue" option. Please add a comment here, and make the annotation clearly say that the presubmits FAILED.
| ) | ||
| return 1 | ||
|
|
||
| # Look up the newest and previous image of every family before changing anything, |
There was a problem hiding this comment.
This undo is incomplete. Making the image active again does not change the running agents. Please also print ./create_instances.py <groups> here.
| decision="$$(buildkite-agent meta-data get vm-image-decision)" | ||
| if [ "$$decision" = rollback ]; then | ||
| ./rollback_vm_image.py "${ROLLBACK_FAMILIES}" && ./create_instances.py "${ROLLBACK_GROUPS}" | ||
| exit 1 |
There was a problem hiding this comment.
Nit: please add a comment that exit 1 stops the pipeline on purpose after a rollback. Also, if create_instances.py fails, the image is already deprecated but the agents still run the new image. Please log this case clearly.
|
|
||
| Used by pipelines/vm-image-rollback.yml when image presubmits fail, and can be | ||
| run manually, e.g. if a problem shows up after promotion to prod: | ||
|
|
There was a problem hiding this comment.
Optional: this repeats the family-to-group mapping in instances.yml (image_family). Reading it from that file would prevent the two from drifting apart.
|
Sorry, my agent identified a few things after I approve.. |
No Problem. I will address them in a follow up PR :) |
We build new VM images for the CI agents and test them with the Bazel presubmits before promoting them. Until now, if those presubmits failed, someone had to figure out by hand how to switch the agents back to the previous image.
With this change, a failed image test pauses the pipeline and asks what to do:
Choosing rollback marks the new image as deprecated, so new VMs boot the previous image again, and recreates the affected agents. The new image isn't deleted, and the undo command is printed, so a rollback is easy to reverse.
I tested the decision flow in bazel-testing with a dry-run copy of the rollback step.