fix(cli): gate bakes on free space, and always remove the staging dir - #316
Merged
Merged
Conversation
Both halves of one failure mode: a bake that runs out of space does not stop, it produces a *corrupt* artifact — a rootfs whose package files carry other files' bytes, or a truncated `memory.bin`. Inside a guest that surfaces much later as `uname: option requires an argument`, `Exec format error`, or `EBADMSG` on a `/var/lib/dpkg` entry, so it reads like a broken build rather than a broken image. - `forkd parent build` (conversion) and `forkd snapshot` (bake) check free space on the target filesystem before the first write and refuse below a 5 GiB reserve (`FORKD_MIN_FREE_GIB` overrides). Conversion is the first multi-GB write of the pipeline, and the snapshot dir holds the clone plus a fresh `memory.bin`. The probe stays advisory: if `statvfs` cannot run, warn and continue rather than block work on a broken measurement. `doctor::available_bytes(path)` is extracted from the check's hardcoded-path statvfs so both callers share one probe. - `forkd snapshot` removes its staging dir on every exit. Only the success path did, so any `?` after the volatile artifacts were written — boot timeout, snapshot error, publish error, interrupt — left a fully written `memory.bin` (GBs) beside the snapshot dir that nothing ever collected. Tests cover the guard (removal, and tolerance of an already-removed dir — the success path renames files out and a stale sweep may beat it) and the probe's ancestor walk, which is what makes it usable before the snapshot dir exists. Signed-off-by: jrimmer <jason@rimmer.net>
Parallel tests write to the same filesystem between the two calls, so exact equality flakes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # CHANGELOG.md
WaylandYang
marked this pull request as ready for review
September 14, 2026 05:02
WaylandYang
approved these changes
Sep 14, 2026
WaylandYang
left a comment
Contributor
There was a problem hiding this comment.
Reviewed. The preflight and staging guard both look right, and the drop order against rootfs_rollback (rollback unwinds first) is correct.
Two maintainer commits pushed to this branch:
available_bytes_walks_up_to_an_existing_ancestorcompared twostatvfsreadings withassert_eq!. Parallel tests write to the same filesystem between those calls, so it could flake. It now allows drift instead of requiring exact equality.- Merged
dev(after #315) and resolved the CHANGELOG conflict.
Verified on Linux (Ubuntu 22.04, stable Rust in Docker): fmt, clippy -D warnings, and cargo test --all all green, both on its own and together with #314/#315/#322. The daemon restore-path gate you mentioned is welcome as a follow-up.
This was referenced Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two halves of one failure mode. Running out of space partway through a write does not fail cleanly — it leaves a corrupt artifact, and the corruption is invisible to anything that only checks the file still runs.
What that looks like inside a guest, from a bake that converted on a full filesystem:
plus
Exec format erroron the same path,EBADMSGon/var/lib/dpkgand/var/lib/apt/listsdirectory entries, andlibidn2.so.0: cannot dynamically load position-independent executable. A settled rootfs hashed byte-identical to its source image in the same session, so the damage is in the write path, not the source. It reaches CI as a compile error ~40s deep (set -euo pipefail; OS="$(uname -s)"in a dependency's build hook) that reads like a broken build script.Free-space preflight
forkd parent build(conversion) andforkd snapshot(bake) now check free space on the target filesystem before the first write and refuse below a 5 GiB reserve, overridable withFORKD_MIN_FREE_GIB:Conversion is the first multi-GB write of the pipeline (it unpacks a whole image into a fresh ext4) and the snapshot dir holds the clone plus a fresh
memory.bin. The reserve matches the thresholdforkd doctoralready reports as "recommended ≥5 GiB".doctor::available_bytes(path)is extracted fromcheck_snapshot_dir_space's statvfs, which was private and hardcoded to$XDG_DATA_HOME/forkd/snapshots— that is the wrong filesystem to gate--rootfs/--snapshot-rootoverrides against, which is why it takes a path now. The probe stays advisory: ifstatvfscannot run, it warns and continues rather than blocking work on a broken measurement.Not included: the same gate on the daemon's restore path. The controller cannot depend on
forkd-cli, so that needs the helper inforkd-vmmplus a policy for the 507/409 response. Happy to do it as a follow-up if the shape here looks right.Staging dir cleanup
forkd snapshotwrotevmstate,memory.binandsnapshot.jsoninto<tag>.staging-<pid>and removed it only on the success path, so any?after the files were written — boot timeout, snapshot error, publish error, interrupt — left a fully writtenmemory.bin(GBs) beside the snapshot dir with nothing to collect it. AStagingDirGuardnow owns the dir for the whole bake; the block comment claiming "a failure at any of those steps drops the staging dir" was not implemented and now is.The guard holds the path, not a handle, so the successful rename of files out of the dir is unaffected, and the redundant success-path
remove_dir_allis gone.Tests
Guard removal and tolerance of an already-removed dir (the success path may have emptied it, and a stale sweep may beat the guard to it); the probe's ancestor walk, which is what makes it usable before the snapshot dir exists.
cargo fmt --checkclean,cargo clippy --all-targets --all-features -D warningsclean,cargo test -p forkd-cligreen (56 passed, 1 ignored). No KVM run — both changes are filesystem-level.