feat(vmm): give every restored child its own rootfs backing

[jrimmer]

Fixes the defect in #317 — concurrent children of one tag sharing a writable rootfs — and carries the version policy that comes with it.

The mechanism, and why it is this one

Neither released Firecracker has drive_overrides: I checked the v1.16.2 and v1.17.0 binaries, not the source, and neither contains the string. So the load-time override from #5774 is not available in a released build.

What a released Firecracker does have is PATCH /drives/{drive_id} on a restored VM, and it moves the device's storage rather than just its config. Measured: after re-pointing a child's rootfs, a write made inside the guest landed in the new backing and not in the original file.

So each child gets its own reflink clone of the tag's rootfs and is re-pointed at it before it runs:

1. cp --reflink=always <tag>/rootfs.ext4 <child>/rootfs.ext4     # free where the fs supports it
2. PUT   /snapshot/load {…, resume_vm:false}                     -> 204
3. PATCH /drives/rootfs {drive_id, path_on_host:<child backing>} -> 204
4. PATCH /vm {"state":"Resumed"}                                 -> 204

The ordering is load-bearing, and step 2 is why. Loading paused leaves the guest un-executed, so nothing it does at boot — journal replay, /var/log, agent startup — can reach the shared base before the drive is re-pointed. Measured: the base stays byte-identical through boot while the backing changes.

Fail closed. A child whose rootfs cannot be re-pointed is not resumed. It would run against the shared base, which is the defect, so the failure surfaces and the child is dropped (killing its Firecracker) rather than run unsafely.

Two API details worth having: PATCH /drives requires drive_id in the body (without it a 400 reads as a body error, not a capability error), and PUT /drives to add a drive post-restore fails with PCI is not enabled since forkd boots pci=off — irrelevant here, relevant if anyone wants per-child extra devices later.

Evidence

Claim	How it was measured, on a KVM host running v1.17.0
children are isolated	two children wrote the same guest path; each read back its own bytes (child-1: AAA=1 BBB=0, child-2: AAA=0 BBB=1)
the shared base is never written	tag rootfs sha256 identical before and after both children wrote
backings are reclaimed on kill	2 backings → 0 after DELETE
a crash strands nothing	SIGKILL the controller, let systemd restart it: 2 backings → 0
daemon-baked tags are covered	POST /v1/snapshots now writes rootfs into snapshot.json; daemon branches inherit it
the predicate	21 assertions, each confirmed by running it

The crash case is interesting and shaped the sweep. On a restart, systemd kills the unit's cgroup, so the children die with the controller; the reap finds dead processes and correctly prunes rather than kills, so nothing reaches Vm::drop and nothing used to remove the backing — while the watchdog skips the directory because its socket file is still present. A sweep now runs before the reap, when those directories are unowned.

Its gate is process ownership, not file presence: a directory is skipped while any live Firecracker's --api-sock points inside it, and if any live Firecracker cannot be read, the sweep removes nothing. Leaking a backing costs disk; deleting a running VM's backing destroys its disk mid-job, so every ambiguity resolves toward keeping it.

The per-kill reclaim is kept alongside the sweep rather than replaced: it covers a process verified ours and confirmed dead at kill time, which the pre-reap sweep cannot see.

Consequences to plan for

• Firecracker ≥ 1.15 is now required, and snapshots are version-pinned, so an upgrade means re-baking every tag. Documented in CHANGELOG as an upgrade note and in docs/VENDORED-FIRECRACKER.md.
• The vendored fork is needed only for --live. It carries MemBackendConfig::shared, and forkd-vmm sends shared in exactly one place — the memfd path live-fork uses. Everything else runs on stock Firecracker. The doc said the fork was mandatory; it now says what it is actually for.
• The clone was never being attempted. chain::reflink_copy passed 0x40209409 for FICLONE; the kernel dispatches on the whole value including the size field, and that one encodes a 32-byte payload (FICLONERANGE's size), so it answers ENOTTY — which the helper's fallback list reads as "no reflink here" and streams a full copy instead. Measured on ZFS, same pool and source file: 0x40209409 -> ENOTTY, 0x40049409 -> clone. So a 12 GiB rootfs cost a full 12 GiB per child (measured: 12291 MiB of pool growth for one child), and the bake path's "reflink preferred" baseline clone was a full copy as well. This PR corrects the constant, after which both are shared extents on ZFS 2.2+/btrfs/XFS. On a filesystem without cloning it remains a full copy per child, so pool size is a storage decision there.
• Backings live under the work dir, which is std::env::temp_dir()-derived. On a host where /tmp is tmpfs that charges a child's rootfs copy to RAM rather than disk; TMPDIR relocates it, which is how we put ours on the same pool as the snapshot root so reflink has both ends to work with.
• Writable volumes are still shared and are the same defect on /dev/vdb: VolumeSpec.read_only defaults to false and the path is frozen, so children of a tag carrying one all open the same file. This PR does not address it; it should be fixed here or in its own change.

Not covered by the evidence

Both crash runs exercised the cgroup-kill flavour, because the children were spawned without a memory limit and so died with the controller. The other flavour — Firecracker outliving the controller, e.g. children in their own per-VM cgroup — exercises the kill path, which is verified through a plain DELETE but not yet through a crash.

Snapshots with no recorded rootfs keep the previous behaviour; daemon-baked ones did until this PR, which is most of why it is here.

[jrimmer]

Defect found while checking what a consumer needs: branching a restored child is no longer durable

Branch used to be safe because a branch recorded the snapshot's rootfs, which persists. With per-child backings it records the child's own backing — inside a work dir that Drop and the sweep reclaim — so the clone dies with its source.

Measured on a v1.17 host with this branch deployed:

source child:            sb-6aa4f2a3-…
branch it -> tag:        br-clone created
kill the source child
spawn from br-clone:     400, "firecracker API PUT /snapshot/load returned 400:
                               Failed to restore from snapshot …"

Two things make it worse than a stale path. The clone's snapshot.json records no rootfs at all, so snapshot_restore_problem cannot see the missing file and forkd reports the FC load failure rather than "this tag's rootfs is gone" — an opaque 400. And the window is not just "source child exits": any restart whose sweep collects the backing breaks every tag derived from that child.

This matters beyond a corner case: Branch taken from a running sandbox is an ordinary way to clone or snapshot one, so clones would break intermittently depending on when their source was reaped.

The fix belongs here: on branch, materialise the child's rootfs into the new tag — a reflink clone of the backing, recorded as the new tag's rootfs — instead of leaving the vmstate pointing into an ephemeral work dir. That also restores the property that a tag's rootfs is its own, which pack/sidecar and snapshot_restore_problem both assume.

I would not merge this PR without it. I have not written it yet, and I am marking it here rather than leaving it for whoever reads the diff.

[jrimmer]

The clone path was dead: wrong FICLONE ioctl number

While checking what a consumer needed I measured the per-child cost on a ZFS pool and got a full copy — 12291 MiB of pool growth for one 12 GiB child, when both the source and the destination were on the same pool and that pool demonstrably clones (cp --reflink=always on the same files costs nothing).

chain::reflink_copy passes 0x40209409 for FICLONE. The kernel dispatches on the whole ioctl value including its size field, and that value encodes a 32-byte payload — FICLONERANGE's size — so it is answered with ENOTTY:

kernel:  #define FICLONE  _IOW(0x94, 9, int)      -> 0x40049409
on ZFS, same source file and directory:
  ioctl 0x40209409 -> ENOTTY        ioctl 0x40049409 -> clone

ENOTTY is in the list this function treats as "this filesystem does not support reflink", so it falls through to a full streamed copy — on every filesystem, silently, including the ones where cloning works. That is why the number being wrong was invisible: the failure path is the same as the legitimate one.

Two consequences beyond this PR: a restored child cost a full copy of the rootfs (measured above), and the bake path's "reflink preferred" baseline clone was a full copy too. Corrected in this PR; docs/VENDORED-FIRECRACKER.md-adjacent notes are not affected, but the cost model in the PR body is now accurate.

[WaylandYang]

Thanks. Pausing on a baked-in drive and re-pointing it with PATCH /drives before resume is the right mechanism, and the evidence table is exactly what this needed. Not merging yet. Beyond the branch-durability blocker you already flagged, here is what I found.

Already landed separately: the FICLONE fix you described (0x40209409 → 0x40049409) was never pushed to this branch; the diff doesn't touch chain.rs. It is in dev now as #322, credited to you and verified on an XFS reflink=1 loopback (old number → ENOTTY, new → shared extents). Please rebase onto dev.

Blocking

1. Branch durability (your comment): a branch taken from a restored child has to materialise the child's backing into the new tag and record it as that tag's rootfs.
2. No runtime Firecracker version gate. The CHANGELOG says PATCH /drives is accepted but doesn't move storage on FC < 1.15, so on an older binary children silently share the base again. That is the exact corruption this PR fixes, now with no signal. A documented minimum isn't enough; this needs to fail closed. Options: check firecracker --version when the controller/CLI starts, or GET /vm/config after the PATCH and refuse to resume unless the drive's path_on_host is the backing.
3. Read-only rootfs snapshots pay a full copy per child for nothing. The CLI read-only (squashfs) bake records snap.rootfs = cfg.rootfs, and restore clones whenever self.rootfs.is_some(). Every child of a read-only tag now copies the squashfs and PATCHes a drive that can never be written. Snapshot doesn't persist the drive's read-only flag (your ask #2 on #317), so the fix probably starts there: record it at bake and skip backings for read-only drives.
4. Cost on filesystems without reflink. On ext4 (the default on most hosts, including our test box) each child is a full streamed copy of the rootfs. The copies run sequentially in the loop before any load, so fork -n N costs N × rootfs in both disk and spawn latency, with no warning. At minimum: say once per restore that the fallback happened, gate on free space (the available_bytes helper from #316), and run the copies in the per-child threads. Whether ext4 hosts should get an explicit opt-in/refusal instead of silent full copies is a product call worth making in this PR.

Non-blocking

• sweep_stranded_backings scans std::env::temp_dir() for forkd-daemon-*. It matches where the controller creates work dirs only while both use the same TMPDIR, so please derive both from one helper.
• The upgrade note (re-bake every tag when moving to FC ≥ 1.15) is a release-level decision. We'll schedule it with the release rather than let it land silently in dev.

I can't exercise the KVM path on my side (the test host has no KVM), so your measured evidence plus a rerun after these changes will be what we rely on.