feat(memory): use fallocate(PUNCH_HOLE) for guest_memfd discard#5792
Draft
JackThomson2 wants to merge 119 commits into
Draft
feat(memory): use fallocate(PUNCH_HOLE) for guest_memfd discard#5792JackThomson2 wants to merge 119 commits into
JackThomson2 wants to merge 119 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## feature/secret-hiding #5792 +/- ##
=========================================================
- Coverage 82.03% 81.06% -0.98%
=========================================================
Files 277 278 +1
Lines 30093 30821 +728
=========================================================
+ Hits 24688 24984 +296
- Misses 5405 5837 +432
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
JackThomson2
force-pushed
the
sh/support_punch_hole
branch
3 times, most recently
from
fa4fe9d to
eab0a13
Compare
Creating a script to build and install a modified kernel with patches applied. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Adding a new integration test to assert that the kernel build script will succeed. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Adding the secret hiding kernel as a default for the buildkite pipeline, this will mean that PR's made against the branch will now be run with the new secret hiding enabled amis. Some tests have been marked to skip as they are kernel dependent so while we are compiling our kernel in CI these could change again. Signed-off-by: Jack Thomson <jackabt@amazon.com>
To make it easier to track the upstream kernels which may change as we rebase, let's mark kernels newer than 6.12 as next for now to make dashboarding easier. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Addressing a comment to move away from dir stacks in our install scripts. We now store the start directly before we move the build directory and cd back to that explicitly. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Run the kernel build as part of our nightly tests so we can monitor it's success. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Add all required linux host kernel patches required for secret hiding. These are: - Direct map removal patches - make kvm_clock work with direct map removed guest_memfd - v2 of KVM_USERFAULT patches [1] - support for UFFDIO_CONTINUE in guest_memfd VMAs - support for write(2) syscall for guest_memfd Based on kvm/next [1]: https://lore.kernel.org/kvm/20250109204929.1106563-1-jthoughton@google.com/ Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
The patches are in the `patches` subdirectory of `hiding_ci`, so if only patches were added, then the check of "any files with parent directory `hiding_ci`" would be false, and the CI step for testing the build of patches wouldn't actually run. Fix this by updating the check to be "any files where any parent directory is `hiding_ci`", which will also catch patches. Reported-by: Jack Thomson <jackabt@amazon.co.uk> Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
Update the build script to allow us to install the secret hidden kernels onto Amazon Linux 2023 instances. We have to as part of this include a script to download and install ena drivers for the instance to allow us to boot. Signed-off-by: Jack Thomson <jackabt@amazon.com>
The output from the build in x86 is archived so updated the script to support installing either output type from the build Signed-off-by: Jack Thomson <jackabt@amazon.com>
Add an 'apt update' before `apt install`. Otherwise, we might hold an old view of the package versions and installation might fail. Signed-off-by: Babis Chalios <bchalios@amazon.es>
This lint forbids using `..Default::default()` in struct initializers after all fields have already been initialized, but this is a useful pattern if you know you want to add more fields to a struct in a future PR without needing to touch a ton of initializers in unittests again (_heavy foreshadowing_). So silence the paperclip. Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
There's no need to test this through VmResources when it can be tested in isolation. Also, everytime I touch MachineConfig I get confsued by where the hell the tests are, cuz not only are they in a different module, they're also one directory level away. So move the tests into machine_config.rs, where it makes sense to have them. Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
With secret freedom, direct accesses to guest memory from the context of the host kernel are no longer possible. This particularly means that we cannot pass pointers to guest memory to the host kernel anymore (at least if the host kernel tries to GUP them). For these scenarios, introduce a utility decorator struct `MaybeBounce` that can optionally do indirect read and write syscalls on guest memory by first memcpy-ing to firecracker userspace, and passing a pointer to firecracker heap memory into the kernel instead. Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
This is particularly useful for virtio devices, where on-demand allocation of bounce buffers leads to sever performance impacts (~80%) in synthetic throughput tests. Additionally, for virtio devices we can know approximately what the optimal size of a statically allocated bounce buffer is. Allocate bounce buffers on the heap, as trying to even temporarily place a 65k bounce buffer on the stack can lead to stack overflow errors. Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
Add support to our virtio devices to allow userspace bounce buffering of virtio buffers. This is an alternative to swiotlb. Don't implement it for vhost-user-blk and for virtio-block with async engine, because I have no idea how that would even work. Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
Add a utility function for creating a guest_memfd and wrapping it into a `File` object. Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
This will later indicate to Firecracker that guest memory should be backed by guest_memfd. Mark vhost-user and async block engine as incompatible, as I/O will require userspace bounce buffers. For vhost-user-blk, we would need to communicate the need for bounce buffers to the backend somehow, and for the async block engine we would need to somehow keep the bounce buffers around until io_uring finishes requests (which is not impossible, but complicated and not needed for now). Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
If secret freedom is enabled, the guest kernel and potential initrd needs to be loaded via bounce buffer, as we cannot directly do `read` syscalls that target guest memory. Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
Needed because we cannot do I/O straight into secret hidden memory - the host kernel cannot access it. Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
Fall back to kvm_user_memory_region in case the 2 version of the struct isnt supported. Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
Used to plug/unplug memory in virtio-mem device. Signed-off-by: Nikita Kalyazin <kalyazin@amazon.com>
Have the `struct Vm` constructor take an argument to indicate whether the VM should be secret free. Use this to determine the correct vm type for guest_memfd support, and store it inside the VM so that we don't have to pass bools to various functions. Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
If the `secret_free` field of the memory_config is set to true in the /machine-config endpoint, back all memory regions using guest_memfd. For our setup, this means both setting the guest_memfd[_offset] fields in kvm_user_memory_region2, as well as mmaping the guest memory and reflecting this VMA back into the memslot's userspace_addr (which is how kvm internal accesses to guest memory will work for these guest_memfd regions, such as mmio emulation on x86). Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
To take snapshots of secret hidden VMs, we need to bounce guest memory through a userspace buffer. Reuse the `Bounce` wrapper type that is already in use for loading the guest kernel / initrd. Signed-off-by: Patrick Roy <roypat@amazon.co.uk>
Previously this would fail on x86 as we set -e. By setting the || true this means the script will continue. The grubby step next will fail if it failed to find the image. Signed-off-by: Jack Thomson <jackabt@amazon.com>
With adding major fault support in guest_memfd, when a major fault is generated, no page is added in the page cache by the kernel. It means that proactive population with write() will succeed for the entire region but possibly one page (kvmclock on x86). We just need to call UFFDIO_CONTINUE in response to such a major fault because we already populated the faulting page via write() earlier. Signed-off-by: Nikita Kalyazin <kalyazin@amazon.com>
This is because when upgrading the guest kernel from 6.1.155 to 6.1.163, /sys/devices/system/cpu/cpu?/cache/* files disappeared. Signed-off-by: Nikita Kalyazin <kalyazin@amazon.com>
- Drop the avoidance of silent kvm-clock activation failure as Sean commented - Fix a compile error when CONFIG_KVM_SW_PROTECTED_VM=y Signed-off-by: Takahiro Itazuri <itazur@amazon.com>
Add the APF patch series to be built into our SH AMI. Signed-off-by: Jack Thomson <jackabt@amazon.com>
This reverts commit 5417c27. Reverting for now while we investigate the regressions in hotplugging Signed-off-by: Jack Thomson <jackabt@amazon.com>
- Upgrade UFFD patches from RFC to v1 including fixes from Harry, Edward and Mike - Replace Sean's SRCU fix PoC with his RFC Current patch set (based on v6.18): - NUMA support in guest_memfd (from v6.19-rc5) - v10 direct map removal + fixup - x86: configurable TLB flushes after direct map removal - v3 kvmclock - v1 KVM userfault - v7 write syscall - fixup for direct map removal and write to work together - v1 UFFD support with fixups (new: RFC -> v1) - RFC srcu_synchronize optimisation from Sean (new: PoC -> RFC) Signed-off-by: Nikita Kalyazin <kalyazin@amazon.com>
Current patch set (based on v6.18): - NUMA support in guest_memfd (from v6.19-rc5) - v11 direct map removal (new: v10 -> v11) - x86: configurable TLB flushes after direct map removal - v3 kvmclock - v1 KVM userfault - v7 write syscall - fixup for direct map removal and write to work together - v2 UFFD support with fixups - RFC srcu_synchronize optimisation from Sean Signed-off-by: Nikita Kalyazin <kalyazin@amazon.com>
Add APF patches for the kernel Signed-off-by: Jack Thomson <jackabt@amazon.com>
Replace the JSON-based UFFD protocol with length-prefixed bitcode encoding for better performance and type safety. Extract UffdMessageBroker from lib.rs into its own module (uffd_broker.rs) with proper error handling — send_fault_request() returns Result instead of panicking, and the iterator logs errors instead of unwinding. Also adds APF socket support to UFFD handler examples and exitless APF ring buffer structures to the handler side. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Add the synchronous/fallback path for KVM async page faults. When a guest vCPU hits a userfault page and the exitless ring buffer is full (or not configured), KVM exits with KVM_EXIT_MEMORY_FAULT + KVM_MEMORY_EXIT_FLAG_APF. The VMM sends the fault request to the UFFD handler over a Unix socket, issues KVM_APF_OP_ACCEPT so the vCPU can re-enter in a halted state, and processes the handler's reply with KVM_APF_OP_READY to wake the vCPU. Includes KvmAPFReq ioctl definitions, APF handling in vcpu handle_userfault() (gated to x86_64), SharedApfStream type, updated create_vcpus signature, and seccomp rules for the new ioctls. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Add the fast path for async page faults that avoids KVM_RUN exit/ re-entry overhead. Each vCPU gets an ExitlessApfContext backed by a memfd-mapped shared page containing notify and completion ring buffers (32 entries each). KVM writes fault GPAs to the notify ring and signals an eventfd; the UFFD handler resolves the page and writes to the completion ring; KVM drains completions via a workqueue and wakes the halted vCPU. Three fds per vCPU (notify eventfd, completion eventfd, shared page memfd) are passed to the UFFD handler via SCM_RIGHTS over the existing Unix socket. Key design: exitless APF setup and handler unblocking happens BEFORE vCPU state restore, since MSR restore (kvm-clock) triggers userfaults on UFFD-registered memory that the handler must be ready to service. APF capability is detected at runtime — gracefully falls back to synchronous UFFD on kernels without KVM_CAP_ASYNC_PF_USERFAULT. Fix default_vmm() test helper to create a real pipe for _apf_pipe_reader instead of stealing fd 0 (stdin), which caused an IO Safety violation (double-close SIGABRT) during test runs. Signed-off-by: Jack Thomson <jackabt@amazon.com>
GUEST_MEMFD_FLAG_WRITE was added accidentally on a rebase on the booted path. Remove this to fixup booted path. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Previously we never passed the APF socket to the handler so we weren't using APF properly Signed-off-by: Jack Thomson <jackabt@amazon.com>
Add integration and performance tests for APF to compare the perf impact Signed-off-by: Jack Thomson <jackabt@amazon.com>
There were a couple of error on the non exitless path for APF. We saw these with the memory hotplug which flodded the APF buffer. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Holding the spinlock while calling the dequeue method caused massive regressions in the hotplugging tests. It was discovered that the root was the spinlock disabling preempt, this had a knock on effect when calling the dequeue. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Current patch set (based on v6.18): - NUMA support in guest_memfd (from v6.19-rc5) - v12 direct map removal (new: v11 -> v12) - x86: configurable TLB flushes after direct map removal - v3 kvmclock - v1 KVM userfault - v7 write syscall - fixup for direct map removal and write to work together - v2 UFFD support with fixups - RFC srcu_synchronize optimisation from Sean - Tmp RFCv2 Async PF (non-published) Signed-off-by: Nikita Kalyazin <kalyazin@amazon.com>
MADV_DONTNEED is a no-op for MAP_SHARED mappings, which means discard_range() previously did nothing for guest_memfd-backed memory. This prevented virtio-mem unplug and balloon inflate from actually freeing physical pages back to the host when secret_free is enabled. Add a fallocate(FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE) path for MAP_SHARED file-backed regions, which punches holes in the guest_memfd backing file and releases the pages from the page cache. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Now that discard_range() uses fallocate(PUNCH_HOLE) for MAP_SHARED guest_memfd regions, the balloon can properly reclaim memory when secret_free is enabled. Remove the restriction that prevented combining balloon with secret_free. Signed-off-by: Jack Thomson <jackabt@amazon.com>
When secret_free is enabled, guest memory is backed by guest_memfd. Hotplug memory regions are never mapped into the VMM's userspace, so host VmRSS doesn't reflect those pages. Guest-side (total - available) is too noisy because mem_available tracks mem_total closely when the guest is idle, so unplug doesn't produce a reliable delta. Use the FC process's cgroup memory.current, which accounts for kernel pages allocated by guest_memfd and correctly drops after fallocate(PUNCH_HOLE). This gives a monotonic, block-accurate signal for virtio-mem unplug and balloon inflate under secret_free. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Add the secret_free fixture parameter to all balloon functional tests, so they run with both SF_OFF and SF_ON variants. This exercises the fallocate(PUNCH_HOLE) discard path for guest_memfd-backed memory during balloon inflate/deflate. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Now that discard_range() uses fallocate(PUNCH_HOLE) for guest_memfd, and get_resident_memory() uses guest meminfo for secret_free VMs, the RSS decrease assertion in check_hotunplug works correctly for secret_free. Remove the skip. Signed-off-by: Jack Thomson <jackabt@amazon.com>
JackThomson2
force-pushed
the
sh/support_punch_hole
branch
from
eab0a13 to
c60c65a
Compare
After discard_range() punches a hole in a guest_memfd-backed region via fallocate(PUNCH_HOLE), the folios for those pages are released. The userfault bitmap bits for those pages, however, may already be cleared from a previous UFFDIO_COPY (the handler clears the bit on successful copy). With a cleared bit and no folio, the next guest access takes the UFFD MINOR-fault path and the handler calls UFFDIO_CONTINUE, which returns EFAULT because there is no folio to install. Re-set the userfault bitmap bits covering the punched range so the next access takes the MISSING-fault path and the handler re-populates via UFFDIO_COPY. Signed-off-by: Jack Thomson <jackabt@amazon.com>
JackThomson2
force-pushed
the
feature/secret-hiding
branch
2 times, most recently
from
334a264 to
7ea69cd
Compare
JackThomson2
force-pushed
the
feature/secret-hiding
branch
from
da326eb to
8eb7b4c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add support for fallocate(PUNCH_HOLE) this will expand our tests to also include balloon and memory hotplugging tests
...
Reason
...
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.PR Checklist
tools/devtool checkbuild --allto verify that the PR passesbuild checks on all supported architectures.
tools/devtool checkstyleto verify that the PR passes theautomated style checks.
how they are solving the problem in a clear and encompassing way.
in the PR.
CHANGELOG.md.Runbook for Firecracker API changes.
integration tests.
TODO.rust-vmm.MADV_DONTNEED is a no-op for MAP_SHARED mappings, which means
discard_range() previously did nothing for guest_memfd-backed memory.
This prevented virtio-mem unplug and balloon inflate from actually
freeing physical pages back to the host when secret_free is enabled.
Add a fallocate(FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE) path for
MAP_SHARED file-backed regions, which punches holes in the guest_memfd
backing file and releases the pages from the page cache.
Signed-off-by: Jack Thomson jackabt@amazon.com