Persist container exit codes and surface them on ContainerSnapshot - #2090
Open
devops-thiago wants to merge 2 commits into
Open
Persist container exit codes and surface them on ContainerSnapshot#2090devops-thiago wants to merge 2 commits into
devops-thiago wants to merge 2 commits into
Conversation
The runtime reported it and ExitMonitor acted on it, but nothing stored it, so a container that crashed was indistinguishable from one that was asked to stop. ContainerSnapshot carries exitCode and exitedAt now, populated in handleContainerExit. Both are optional, so a snapshot from before this change - or a container that stopped before the apiserver started - decodes fine and simply has no code.
The exit code lived only in the in-memory ContainerState, so it survived exactly as long as the apiserver process did. Restart the engine and a container that had died with 127 was indistinguishable from one stopped on purpose - which is the moment you most want the answer, since you are usually restarting *because* something went wrong. - ExitRecord is written to exit.json in the container bundle when handleContainerExit sees a code, and read back by loadAtBoot into the snapshot it builds. Bundle already owned this directory's layout, so the read/write pair sits there next to the other bundle files. - A failed write is logged, not thrown: the container really has exited, and refusing to record that would leave the state machine wedged. - ContainerStatus carries exitCode/exitedAt too, so `container inspect` shows them. ManagedContainer is what the CLI serializes, and it was dropping both on the floor.
devops-thiago
force-pushed
the
exit-code-persistence
branch
from
August 7, 2026 13:37
2d29c20 to
898e4da
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion issue: #1501. Overlaps with draft #1503 — see the comparison below.
Type of Change
Motivation and Context
ContainerSnapshotexposesRuntimeStatus.stoppedbut not the exit code, so an orchestrator implementingdepends_on: condition: service_completed_successfullycannot distinguish clean exit from failure — the motivation laid out in #1501.This implementation surfaces the code and also persists it in the container bundle, restoring it at bootstrap. That second half matters because the API server does not run forever: it is restarted by upgrades, crashes, and reboots, and an in-memory-only exit code reads as
nilafterwards — precisely when an orchestrator polling a one-shot container needs the answer most.Description
Two commits, reviewable independently:
ContainerSnapshotgainsexitCode: Int32?andexitedDate: Date?, populated from the existingExitMonitorcallback at the stop transition, and carried throughContainerStatussocontainer inspectshows them.exit.json, atomic write) when the container stops, loaded during bootstrap for containers found stopped, and removed on delete with the bundle.Relationship to draft #1503: same goal and compatible shape for the snapshot field; that draft is snapshot-only, so the code is lost whenever the API server restarts. If the author or maintainers prefer to land #1503 first, the persistence half here rebases cleanly on top of it — either path gets orchestrators a durable answer, which is the property worth having.
We run both commits in production in a GUI embedder of this engine, where the app relaunching the engine is an everyday event rather than an edge case.
Testing
swift test --filter "ContainerResourceTests|ContainerAPIServiceTests"— 83 tests pass on this branch at abff418.container inspect, the engine is restarted, and the same code is still reported.