Skip to content

fix(cli): gate bakes on free space, and always remove the staging dir - #316

Merged
WaylandYang merged 3 commits into
deeplethe:devfrom
jrimmer:fix/bake-space-and-staging
Sep 14, 2026
Merged

WaylandYang merged 3 commits into
deeplethe:devfrom
jrimmer:fix/bake-space-and-staging

Conversation

@jrimmer

@jrimmer jrimmer commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Two halves of one failure mode. Running out of space partway through a write does not fail cleanly — it leaves a corrupt artifact, and the corruption is invisible to anything that only checks the file still runs.

What that looks like inside a guest, from a bake that converted on a full filesystem:

$ uname --version
uniq (GNU coreutils) 9.1          # /usr/bin/uname is a valid ELF holding another file's bytes
$ uname -s
uname: option requires an argument -- 's'

plus Exec format error on the same path, EBADMSG on /var/lib/dpkg and /var/lib/apt/lists directory entries, and libidn2.so.0: cannot dynamically load position-independent executable. A settled rootfs hashed byte-identical to its source image in the same session, so the damage is in the write path, not the source. It reaches CI as a compile error ~40s deep (set -euo pipefail; OS="$(uname -s)" in a dependency's build hook) that reads like a broken build script.

Free-space preflight

forkd parent build (conversion) and forkd snapshot (bake) now check free space on the target filesystem before the first write and refuse below a 5 GiB reserve, overridable with FORKD_MIN_FREE_GIB:

refusing to bake a snapshot: only 1.2 GiB free on the filesystem holding /var/lib/forkd/snapshots (need 5.0 GiB).

Conversion is the first multi-GB write of the pipeline (it unpacks a whole image into a fresh ext4) and the snapshot dir holds the clone plus a fresh memory.bin. The reserve matches the threshold forkd doctor already reports as "recommended ≥5 GiB".

doctor::available_bytes(path) is extracted from check_snapshot_dir_space's statvfs, which was private and hardcoded to $XDG_DATA_HOME/forkd/snapshots — that is the wrong filesystem to gate --rootfs/--snapshot-root overrides against, which is why it takes a path now. The probe stays advisory: if statvfs cannot run, it warns and continues rather than blocking work on a broken measurement.

Not included: the same gate on the daemon's restore path. The controller cannot depend on forkd-cli, so that needs the helper in forkd-vmm plus a policy for the 507/409 response. Happy to do it as a follow-up if the shape here looks right.

Staging dir cleanup

forkd snapshot wrote vmstate, memory.bin and snapshot.json into <tag>.staging-<pid> and removed it only on the success path, so any ? after the files were written — boot timeout, snapshot error, publish error, interrupt — left a fully written memory.bin (GBs) beside the snapshot dir with nothing to collect it. A StagingDirGuard now owns the dir for the whole bake; the block comment claiming "a failure at any of those steps drops the staging dir" was not implemented and now is.

The guard holds the path, not a handle, so the successful rename of files out of the dir is unaffected, and the redundant success-path remove_dir_all is gone.

Tests

Guard removal and tolerance of an already-removed dir (the success path may have emptied it, and a stale sweep may beat the guard to it); the probe's ancestor walk, which is what makes it usable before the snapshot dir exists.

cargo fmt --check clean, cargo clippy --all-targets --all-features -D warnings clean, cargo test -p forkd-cli green (56 passed, 1 ignored). No KVM run — both changes are filesystem-level.

jrimmer and others added 2 commits September 11, 2026 13:54
Both halves of one failure mode: a bake that runs out of space does not
stop, it produces a *corrupt* artifact — a rootfs whose package files
carry other files' bytes, or a truncated `memory.bin`. Inside a guest
that surfaces much later as `uname: option requires an argument`,
`Exec format error`, or `EBADMSG` on a `/var/lib/dpkg` entry, so it
reads like a broken build rather than a broken image.

- `forkd parent build` (conversion) and `forkd snapshot` (bake) check
  free space on the target filesystem before the first write and refuse
  below a 5 GiB reserve (`FORKD_MIN_FREE_GIB` overrides). Conversion is
  the first multi-GB write of the pipeline, and the snapshot dir holds
  the clone plus a fresh `memory.bin`. The probe stays advisory: if
  `statvfs` cannot run, warn and continue rather than block work on a
  broken measurement. `doctor::available_bytes(path)` is extracted from
  the check's hardcoded-path statvfs so both callers share one probe.

- `forkd snapshot` removes its staging dir on every exit. Only the
  success path did, so any `?` after the volatile artifacts were written
  — boot timeout, snapshot error, publish error, interrupt — left a
  fully written `memory.bin` (GBs) beside the snapshot dir that nothing
  ever collected.

Tests cover the guard (removal, and tolerance of an already-removed dir —
the success path renames files out and a stale sweep may beat it) and the
probe's ancestor walk, which is what makes it usable before the snapshot
dir exists.

Signed-off-by: jrimmer <jason@rimmer.net>
Parallel tests write to the same filesystem between the two calls, so
exact equality flakes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@WaylandYang
WaylandYang marked this pull request as ready for review September 14, 2026 05:02

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. The preflight and staging guard both look right, and the drop order against rootfs_rollback (rollback unwinds first) is correct.

Two maintainer commits pushed to this branch:

  • available_bytes_walks_up_to_an_existing_ancestor compared two statvfs readings with assert_eq!. Parallel tests write to the same filesystem between those calls, so it could flake. It now allows drift instead of requiring exact equality.
  • Merged dev (after #315) and resolved the CHANGELOG conflict.

Verified on Linux (Ubuntu 22.04, stable Rust in Docker): fmt, clippy -D warnings, and cargo test --all all green, both on its own and together with #314/#315/#322. The daemon restore-path gate you mentioned is welcome as a follow-up.

@WaylandYang
WaylandYang merged commit 28fffb0 into deeplethe:dev Sep 14, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants