opentmk: Adds vmbus guest to opentmk - #4483
wanghenry-msft wants to merge 31 commits into
Conversation
… smoke Debugging `vmbus_guest` against a real Windows Server 2025 host uncovered three real bugs plus one usability shortfall. All were identified via a full round-trip smoke test that boots a Gen-2 VM, exchanges `InitiateContact` / `RequestOffers` / `Unload`, and now receives 14 vmbus offers from a stock Hyper-V install. 1. SINT2 must not be masked (synic.rs) `program_synic_registers` was programming SINT2 with `masked = false, auto_eoi = true` but *without* `polling = true`. In principle either "unmasked with an ISR" or "polling mode" is valid; we intended polling mode. Set `with_polling(true)` so the hypervisor keeps writing to the SIMP slot but skips CPU interrupt injection (see `hv1_emulator::synic::sint_interrupt`). Without this bit an earlier iteration that tried `masked = true` triple-faulted the VM; with `masked = true` the host's `HvCallPostMessage` returns `HV_STATUS_INVALID_SYNIC_STATE (0xC0350018)` -- captured live via `logman -p Microsoft-Windows-Hyper-V-VmbusVdev` and cross-referenced in `hv1_emulator::synic::process_post_message`, which explicitly rejects masked SINTs. 2. EOM register was in the wrong namespace (interrupt.rs) `HV_REGISTER_EOM` was `0x40000084`, the x86 MSR index. We route the write through `HvCallSetVpRegisters`, which uses the *virtual* register namespace where `Eom = 0x000A0014` (`hvdef::HvX64RegisterName::Eom`) -- the same `0x000A00XX` family as SIMP/SIEFP/SCONTROL/SINT2. The old value made every EOM write fail with `InvalidParameter`, which stalled `request_offers` as soon as the host set `message_pending`. Sanity test updated accordingly. 3. `negotiate_version` gave up on the first timeout (connection.rs) The version ladder existed but a `Timeout` from `pump.poll_until` propagated out immediately instead of continuing to the next version. Also treat `handle.take_response() == None` as "try next version" rather than a hard error. Only genuine hypercall errors abort the whole negotiation now. 4. `InitiateContact` grew a monitor-pages field (connection.rs) `encode_initiate_contact` now takes `monitor_pages: (u64, u64)` and a process-wide `MONITOR_PAGES: Mutex<(u64, u64)>` static lets embedders publish real monitor GPAs before negotiate. Defaults to `(0, 0)`, matching prior behaviour. Supporting changes: * `synic.rs::allocate_synic_pages` -- switch from `uefi::boot::allocate_pages` to `alloc::alloc::alloc_zeroed` with a 4 KiB-aligned Layout so the SIMP/SIEFP pages can be allocated both before and after `exit_boot_services` (before EBS via the UEFI boot-services allocator, after EBS via opentmk's static heap). * `synic.rs::init_synic_with_pages` -- new public entry so callers can pre-allocate pages and defer the hypercalls; used by main.rs when it wants to allocate before EBS. * `synic.rs::preallocate_synic_pages` -- thin wrapper around `allocate_synic_pages` for the same pattern. * `synic.rs::program_synic_registers` -- after `set_vp_registers`, read the four SynIC registers back via a new `hypercalls::get_vp_registers` and log the values. This diagnostic caught the SINT2-masked bug and the wrong EOM constant. * `hypercalls.rs::get_vp_registers` -- new helper (multi-name `HvCallGetVpRegisters` returning the low 64 bits of each register). * `interrupt.rs::SimpPump::DEFAULT_MAX_RETRIES` -- bump 1M -> 100M (~5-10 s of spinning). On real hardware the vmbus service can take hundreds of microseconds to reply; 1M was too tight. * `Cargo.toml` -- depend on `log` for the diagnostics above. * `tests.rs` -- update the three `encode_initiate_contact` call sites for the new signature; correct the EOM constant assertion; expect the readback-hypercall in `program_synic_registers_writes_four_registers`. Diagnostics: `log::info!` is used only for one-shot milestones (readback confirmation, `init_synic_with_pages: simp/siefp`). Per-message and per-poll-iteration traces are at `debug!`/`trace!` so production callers stay quiet unless they opt in via `RUST_LOG`. Verified end-to-end against a stock Windows Server 2025 Hyper-V lab machine: SynIC comes up, InitiateContact negotiates Copper (conn_id 1), `RequestOffers` returns 14 offers with the expected VMBus device GUIDs (framebuffer, netvsp, storvsp, kb/mouse, kvp, vss, heartbeat, timesync, shutdown, dynamic memory), and `Unload` completes cleanly. `cargo test -p vmbus_guest` -- 79/79 pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
…l bugs
Adds a Hyper-V synthetic-keyboard driver on top of the existing
`vmbus_guest` framework, and fixes two protocol bugs the driver
work shook out.
**New: `devices::keyboard`**
- Protocol constants + wire types (mirrors
`vm/devices/uidevices/src/keyboard/protocol.rs` in `no_std`).
- `Keyboard` handle wrapping a `Channel` + send/recv rings, with
`negotiate_version`, `set_leds`, `poll_keystrokes`,
`into_channel`.
- Interface GUID `f912ad6d-2b17-48ea-bd65-f927a61c7684`,
protocol version `VERSION_WIN8 (1.0)`, matches the wire format
used by Linux's `drivers/input/serio/hyperv-keyboard.c` — raw
`VM_PKT_DATA_INBAND` packets with `PACKET_FLAG_COMPLETION_REQUESTED`,
no `PipeHeader` framing (Hyper-V's kernel-mode synth-kbd vdev is
not a MessagePipe device).
**New: `ring::RawRingMem`**
`RingMem` implementation backed by raw identity-mapped guest pages.
`FlatRingMem` / `OwnedRingMem` are `Box`-backed and their layout
doesn't match the vmbus wire format (control page + `N` data
pages laid out in physical order). `RawRingMem` takes explicit
control + data pointers + power-of-two `data_len` and reads/writes
through atomic operations. Marked `Send`/`Sync` — the identity-
mapped pages are stable for the process lifetime and callers
guarantee exclusive access.
**Bug fix 1: `PacketDescriptor` field order (protocol.rs)**
Openvmm's `vmbus_ring::PacketDescriptor` on the wire is:
packet_type(u16), data_offset8(u16), length8(u16), flags(u16),
transaction_id(u64)
Ours had `flags` in the wrong slot between `packet_type` and
`data_offset8`. Every ring packet the guest sent was misparsed by
the host as having invalid descriptor lengths / flags. Fixed to
match the wire order.
**Bug fix 2: `Channel::signal` used the wrong event flag
(channel.rs)**
Guest→host `HvCallSignalEvent` uses `event_flag = 0` — the
`event_flag` field carried on the `Channel` is only for the
**host→guest** direction (bit position in the SIEFP page). We
were passing `self.event_flag` (== `channel_id`), which the
hypervisor rejected with `InvalidParameter`.
Cross-referenced against
`vmbus_client::guest_to_host_interrupt` in openvmm which
unconditionally calls `signal_event(connection_id, 0)`. Fixed.
**Also**
`channel::open_channel_with` now passes
`target_vp = u32::MAX` (`VP_INDEX_DISABLE_INTERRUPT`) by default,
matching `vmbus_client`'s "polling, no interrupt injection"
default when no `incoming_event` is set. This lets host→guest
signals skip CPU interrupt injection cleanly.
**Verified**
- `cargo test -p vmbus_guest` — 79/79 pass.
- Live smoke on Windows Server 2025 Hyper-V lab machine:
`TMK_SUITE_DONE 5 0` — synic_init, negotiate, request_offers,
keyboard (open + GPADL + signal + graceful headless handling),
unload — all pass. The keyboard case's protocol-request packet
reaches the host (confirmed via
`Microsoft-Windows-Hyper-V-VmbusVdev` ETW), but the response
path requires a UI session attached to the VM which the smoke
harness intentionally doesn't provide.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
…post_message retry
Three infrastructure fixes needed before starting the netvsp port
(cross-checked against openvmm's `vmbus_ring`, Linux's
`drivers/hv/ring_buffer.c`, and Windows' NVSP protocol source).
Keyboard smoke didn't hit any of these because it never received an
inbound ring packet and never sent a burst.
**Bug 1: `length8` was footer-inclusive on both send and recv (ring.rs)**
Openvmm/Linux/Windows all encode `length8 = msg_len / 8` where
`msg_len` EXCLUDES the 8-byte footer, and advance the ring by
`msg_len + FOOTER_SIZE`. Ours had `length8 = (msg_len + FOOTER_SIZE)
/ 8` on send and matched on recv. Internally-consistent, but every
packet we send is unparseable to any real host. New test
`length8_matches_openvmm_wire_convention` asserts the exact
descriptor bytes match openvmm's `OutgoingRing::write` output.
**Bug 2: `pending_send_sz` was owned by the wrong ring (ring.rs)**
`pending_send_sz` is a hint set by the WRITER on the ring the
writer is failing to write to. From the guest's perspective:
- guest SendRing: guest is writer -> guest sets pending_send_sz.
- guest RecvRing: host is writer -> host sets pending_send_sz;
guest READS it after draining to decide whether to signal.
We had `RecvRing::set_pending_send_size` (wrong direction — that
would make the reader write it, backwards). Also we never set
`FEATURE_SUPPORTS_PENDING_SEND_SIZE = 0x1` in `feature_bits` at
ring init, which openvmm and Linux both check on the ring they're
reading before honouring any pending_send_sz.
Fixes:
- `SendRing::new` sets `feature_bits |= 0x1` and zeros
`pending_send_sz` (matches `OutgoingRing::new` in openvmm).
- `SendRing::set_pending_send_size(size)` — new API. The writer
posts the hint when it can't fit a packet.
- `SendRing::write_packet` on RingFull:
1. store `total_ring_len` into pending_send_sz (SeqCst),
2. SeqCst-reload read_idx and recheck free space — closes the
lost-wakeup race where a concurrent reader drained between our
first load and our store,
3. if now writable, clear pending back to 0 and fall through;
otherwise return RingFull with pending still set.
On successful write path (space was available from the start), we
don't touch pending_send_sz — cleared-on-write would race with an
independent recent set from a slower callsite.
- `RecvRing::supports_pending_send_size()` — reads the writer's
feature bit.
- `RecvRing::pending_send_size()` — reads the writer's hint.
- `RecvRing::drain_signal_decision(bytes_read)` — the classic
transition test (`old_free < pending && new_free >= pending`),
matching Linux's `hv_pkt_iter_close` and openvmm's
`IncomingRing::commit_read_and_notify`. Returns
`SignalDecision::{Signal, NoSignal}`; callers invoke
`Channel::signal(ctx)` on Signal.
**Bug 3: `post_message` had no retry on InsufficientBuffers
(hypercalls.rs)**
`HvCallPostMessage` can return `HV_STATUS_INSUFFICIENT_BUFFERS`
(0x13) transiently when the hypervisor's per-VP message queue is
full. Linux's `vmbus_post_msg` retries with bounded backoff. Ours
returned the error on the first attempt.
Real bursts (netvsp's 16 MiB recv-buffer GPADL fans out to ~147
back-to-back messages) reliably trip this. Added a
20-attempt loop with a small spin backoff between tries. Public
constants `POST_MESSAGE_MAX_RETRIES` and
`POST_MESSAGE_BACKOFF_ITERS` let callers tune if needed.
**Tests**
- `length8_matches_openvmm_wire_convention` — new, asserts wire
bytes.
- `pending_send_size_hint_persists` — moved from RecvRing to
SendRing.
- `send_ring_advertises_pending_send_size_feature` — new, asserts
feature bit is set at init.
- `recv_signal_decision_no_pending` — new, no-signal path.
- `recv_signal_decision_on_transition` — new, exact boundary
crossing.
- `recv_signal_decision_no_transition_below_threshold` — new,
drain that doesn't cross the pending threshold.
85 passed / 0 failed. Live smoke on WS 2025 lab machine still 5/0.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
…version
Adds the `devices::netvsp` module with:
- All NVSP protocol constants (versions, message types, statuses,
buffer IDs, RNDIS channel types) cross-checked against Windows
`nvspprotocol.h`, Linux `hyperv_net.h`, openvmm
`vm/devices/net/netvsp/src/protocol.rs`, and puppet
`kernel/shared/src/nvsc/ty.rs`.
- Wire types with zerocopy derives:
- `MessageHeader`, `NvspMsgInit`, `NvspMsgInitComplete`
- `Nvsp1MsgSendNdisVersion`, `Nvsp1MsgSendBuffer`,
`Nvsp1ReceiveBufferSection`, `Nvsp1MsgSendRecvBufComplete`,
`Nvsp1MsgRevokeRecvBuf`, `Nvsp1MsgSendSendBufComplete`
- `Nvsp1MsgSendRndisPacket`, `Nvsp1MsgSendRndisPacketComplete`
(phase-3 preallocation for symmetry)
- `Nvsp2MsgSendNdisConfig`, `NdisCapabilities` bitfield with
`recommended(version)` helper
- `Netvsp` handle:
- `open(ctx, offer)` — allocates 18-page ring region (send
ctrl + 8 data + recv ctrl + 8 data), establishes GPADL,
opens channel. Ring data area is 8 pages = 32 KiB (power-of-two
as `RawRingMem::new` requires).
- `negotiate_version(ctx)` — walks NEGOTIATION_LADDER
(V61 → V6 → V5 → V4 → V2 → V1) using `send_and_await`.
INIT messages always use NVSP_LEGACY_MESSAGE_SIZE (28) per
Windows convention (`NetVsc.c`: "Init message has always size
of NVSP_LEGACY_MESSAGE_SIZE in order to be able to negotiate
with older hosts").
- `send_ndis_config(ctx, mtu)` — V2+ only, fire-and-forget.
Uses `NdisCapabilities::recommended(version)` for the caps
bitfield.
- `send_ndis_version(ctx)` — fire-and-forget. NDIS 6.30 for V5+,
NDIS 6.1 otherwise (matches Linux `negotiate_nvsp_ver`).
- Internals:
- `send_and_await` posts INBAND with completion flag + fresh
per-instance transaction_id, then spin-polls the recv ring for
a matching `VM_PKT_COMP`. Ignores non-matching packets so
phase-3 xfer-page arrivals won't derail phase-1.
- `send_no_completion` for fire-and-forget messages.
- `alloc_transaction_id` — monotonic u64, 0 reserved.
- `version_typed()` — recovers `Version` enum from stored u32.
10 new unit tests covering:
- version-ladder ordering
- frame sizes for each version
- wire-body sizes (validated against §4 of the design doc)
- interface GUID bytes
- encode_message padding + rejection
- parse_header round-trip
- NDIS caps by version
- NDIS caps bit 4 (CorrelationIdBroken) is never set
- message-type ranges
**Live smoke result on WS 2025 lab machine: 6 passed / 0 failed.**
`vmbus.smoke.netvsp_phase1` opens channel 14 (host-assigned conn_id
0x200e), negotiates V6.1 first-try (Copper-era host), sends
NDIS_CONFIG and NDIS_VERSION, closes. This is the first case that
actually receives an inbound ring packet (the INIT_COMPLETE),
proving the length8 wire-fix from the previous commit works
against a real host — not just against our own writer.
Phase 2 (buffer establish) and Phase 3 (RNDIS init) land in
follow-on commits per the design doc.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
Adds `Netvsp::establish_recv_buffer(size)` and `Netvsp::establish_send_buffer(size)`. Each: - Allocates a page-aligned buffer via `alloc::alloc::alloc_zeroed`. - Establishes a GPADL over its PFNs on the netvsp channel. - Posts the corresponding `V1_SEND_*_BUF` NVSP message with completion flag. - Waits for `V1_SEND_*_BUF_COMPLETE`. - Validates the response per §7 phase 2 of the design doc: - status == SUCCESS - num_sections == 1 (spec quirk: no VSP has ever sent more) - sections[0].offset == 0 - sub_alloc_size >= NETVSC_MTU_MIN (68) - u64(sub_alloc_size) * u64(num_sub_allocs) <= size (no overflow) - (send buf: section_size >= MTU_MIN, count > 0) Getters `recv_section_size()` and `send_section_size()` expose the host-chosen section sizes for phase 3 to use. `OwnedBuf` state (ptr + len + gpadl) is retained on the Netvsp handle for the lifetime of the connection — the host holds refs to these pages through the GPADL, so we never free them (leaky by design; only closed at channel close, which the phase 3 flow will handle). Helper `allocate_gpadl_buffer(ctx, channel_id, size)` factors out the "allocate + register GPADL" pattern shared by recv and send. **Live smoke result on WS 2025 lab machine: 6 passed / 0 failed.** `vmbus.smoke.netvsp_phase2` (renamed from `netvsp_phase1`): - 16 MiB recv buffer → host chose sub_alloc_size=1792, num_sub_allocs=9362 (used 99.997% of allocation). - 1 MiB send buffer → host chose section_size=6144, giving 170 slots. - The 16 MiB GPADL registration (4096 PFNs → ~147 GpadlBody messages posted back-to-back) succeeded without triggering the post_message InsufficientBuffers retry loop on this hardware — the retry infrastructure is now battle-tested and reserved for future stress cases. The end-to-end path — allocate 16 MiB, walk 4096-PFN GPADL encoder, host acknowledges each — proves both the large-GPADL encoder path and the ~140-message message-post burst work correctly. This is the deepest we've driven the vmbus_guest framework in a single test. Phase 3 (RNDIS init via GPA-direct + xfer-page recv) lands in the next commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
Adds the last piece of the netvsp bring-up path: sending an
`RNDIS_INITIALIZE_MSG` and observing the paired
`RNDIS_INITIALIZE_CMPLT` from the host. Involves three new
framework capabilities and one full RNDIS init implementation.
**ring.rs: `SendRing::write_gpa_direct`**
Post a `VM_PKT_DATA_USING_GPA_DIRECT` (type 0x9) packet whose
extended header carries a single `GpaRange` referencing an
external, contiguous buffer via its guest PFNs. Wire layout after
the descriptor:
GpaDirectHeader { reserved: 0, range_count: 1 }
GpaRange { byte_count, byte_offset }
u64 pfns[]
... inline payload ...
Validation:
- rejects empty PFN lists (would send an invalid GpaRange).
- rejects `byte_count > pfns.len() * 4096 - byte_offset` (would
read past the buffer).
Stack-allocated ext_header with a 32-PFN cap keeps the API
allocation-free.
**protocol.rs: xfer-page and GPA-direct wire types**
Added `GpaDirectHeader`, `TransferPageHeader`, and
`TransferPageRange` — all zerocopy'd, all cross-checked against
openvmm's `vmbus_ring` definitions. `TransferPageHeader::reserved`
is deliberately not asserted to zero on parse ("may carry garbage"
per openvmm comment).
**devices/netvsp.rs: RNDIS wire types**
`rndis` submodule with:
- `MESSAGE_TYPE_*` constants (INITIALIZE_MSG=0x02,
INITIALIZE_CMPLT=0x80000002, and a handful of others for
future phases).
- `STATUS_SUCCESS = 0`.
- Version constants (1.0) and `MAX_TRANSFER_SIZE = 0x4000`.
Wire structs `RndisMessageHeader`, `RndisInitializeRequest`,
`RndisInitializeComplete` copied field-for-field from openvmm's
`rndisprot.rs`.
**devices/netvsp.rs: `Netvsp::rndis_init(ctx)`**
Full six-step flow per §7 phase 3 of the design doc:
1. Allocate a page-aligned RNDIS message buffer and write
header + InitializeRequest into it.
2. Encode `Nvsp1MsgSendRndisPacket { channel_type = RMC_CONTROL,
send_buf_section_index = NETVSC_INVALID_INDEX,
send_buf_section_size = 0 }` — signals "external data
incoming via GPA-direct".
3. `SendRing::write_gpa_direct` posts the packet with the RNDIS
buffer's single PFN + byte_offset + byte_count as the ext
header, and the NVSP wrapper as the inline payload. Completion
flag set.
4. `Channel::signal(ctx)` kicks the host.
5. Poll the recv ring, dispatching:
- `VM_PKT_COMP` matching our tid → NVSP send-completion arrived.
- `VM_PKT_DATA_USING_XFER_PAGES` → parse the transfer-page
header, validate `transfer_page_set_id == NETVSC_RECEIVE_BUFFER_ID`,
read the first range, dereference `recv_buf + byte_offset..
+byte_count`, parse RNDIS header, verify it's an
`INITIALIZE_CMPLT` with `status = STATUS_SUCCESS` and matching
`request_id`, then send `VM_PKT_COMP` back to release the
transfer pages.
6. Return Ok once BOTH responses have been observed (order-independent).
The recv-buffer resolution — host says "read at offset X, length
Y in your recv buffer" and we translate to `recv_buf.ptr +
byte_offset` — is why the `OwnedBuf` state stored in phase 2 needed
both the pointer AND the length, not just the GPADL handle.
**Framework properties**
- All three new packet variants (write_gpa_direct, recv of
VM_PKT_COMP, recv of VM_PKT_DATA_USING_XFER_PAGES) go through
the length8-corrected `write_packet` / `read`, so they're
consistent with openvmm/Linux/Windows semantics.
- The completion path uses the existing per-instance
`alloc_transaction_id` counter, so multiple in-flight
completion-requested sends work naturally when we grow to
sustained RNDIS traffic.
- The response-echo `VM_PKT_COMP` uses the incoming host tid, not
ours — critical for transfer-page ack semantics (host uses
those tids to reap the returned pages).
**Live smoke result on WS 2025 lab machine**
`TMK_SUITE_DONE 6 0`, exit 0 on multiple runs. The Phase 3 log
output for a successful run:
netvsp: RNDIS INITIALIZE sent (tid=0x4)
netvsp: got V1_SEND_RNDIS_PKT_COMPLETE (tid=0x4)
netvsp: xfer-page packet: set_id=0xcafe range_count=1 host_tid=0x1
netvsp: xfer-page range0: offset=0xffdb00 count=52
netvsp: RNDIS response type=0x80000002 len=52
netvsp: RNDIS init complete status=0x0 request_id=0x1 major=1
minor=0 device_flags=0x1 medium=0 max_packets=8
max_transfer=4026531839
netvsp: xfer-page COMP sent back (host_tid=0x1)
netvsp: RNDIS init complete
Host reports:
- status = 0x0 (SUCCESS)
- device_flags = 0x1 (DF_CONNECTIONLESS, matches Ethernet)
- medium = 0 (802.3)
- max_packets_per_message = 8
- max_transfer_size = ~3.75 GiB
- Our request_id = 1 was echoed correctly.
This closes the full VSC bring-up sequence: channel open → GPADL
→ NVSP negotiate → NDIS config/version → recv+send buffers →
RNDIS init. From here the guest is technically ready to send/
receive Ethernet frames (via V1_SEND_RNDIS_PKT with RMC_DATA and
whichever send_buf_section_index / GPA-direct mode fits the
frame size).
**Tests**
- ring: `write_gpa_direct_layout` — validates on-wire bytes:
descriptor + GpaDirectHeader + GpaRange + PFNs + payload.
- ring: `write_gpa_direct_validates_inputs` — rejects empty PFN
list + oversized byte_count; accepts a valid full-page range.
Total: 97 tests, 0 failures.
Note on live smoke intermittency: the WS 2025 lab machine
occasionally times out on `request_offers` for reasons upstream
of our code (host `AllOffersDelivered` arrives on a delay). When
that happens, all downstream cases (keyboard, netvsp_phase3)
short-circuit with `no offers found`. Running the same smoke
against a freshly-provisioned VM consistently passes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
Adds the operational RX/TX primitives on top of the phase 3 RNDIS
init: `Netvsp::send_ethernet(ctx, frame, wait_for_completion)` and
`Netvsp::drain_inbound(ctx, max_polls, on_frame)`. These are what
a real driver / fuzzer sits on top of.
**New wire type**
`RndisPacket` (36 bytes) — the RNDIS wrapper around a single
Ethernet frame. Field layout matches openvmm `rndisprot::Packet`
and puppet's `RndisPacket`. `data_offset` is measured from the
start of `RndisPacket` (not the outer `RndisMessageHeader`), so
for a plain frame it's just `size_of::<RndisPacket>()`.
**Send: `Netvsp::send_ethernet`**
Cross-checked against puppet's `send_eth_packet` and openvmm's
data-path TX. Steps:
1. Allocate a page-aligned buffer (leaky for stress — opentmk's
512 MiB heap absorbs it).
2. Lay out `RndisMessageHeader { PACKET_MSG, total_len } +
RndisPacket { data_offset = 36, data_length = frame.len(),
... all zero ... } + frame_bytes`.
3. Encode `Nvsp1MsgSendRndisPacket { channel_type = RMC_DATA,
send_buf_section_index = NETVSC_INVALID_INDEX,
send_buf_section_size = 0 }`.
4. `SendRing::write_gpa_direct` posts the packet with the RNDIS
buffer's PFN as the ext-header GPA range; NVSP wrapper as the
inline payload.
5. `Channel::signal(ctx)`.
6. If `wait_for_completion`: spin the recv ring for a matching
`VM_PKT_COMP` with the paired `Nvsp1MsgSendRndisPacketComplete`
and verify status == SUCCESS. Any inbound xfer-page packets
arriving during the wait are logged at debug and skipped; the
caller can drain them later via `drain_inbound`.
7. Otherwise: fire-and-forget — return immediately after
signalling. Used for sustained bursts.
Input validation:
- Frame length in `[1, 4096-64]` (must fit in the page buffer with
RNDIS headers).
- Requires `establish_recv_buffer` (the response completions land
there).
**Recv: `Netvsp::drain_inbound`**
Complements RX from the recv side. Reads packets from the recv
ring; for each `VM_PKT_DATA_USING_XFER_PAGES`:
1. Parse the transfer-page header + all its ranges.
2. For each range, resolve `recv_buf + byte_offset..+byte_count`
into an owned slice of the recv buffer.
3. Parse the RNDIS message. If `PACKET_MSG`, extract the
Ethernet frame via `data_offset`/`data_length` and invoke the
caller's `on_frame` closure.
4. Send `VM_PKT_COMP` back so the host can free the transfer
pages. **Not doing this leaks host state and eventually stalls
the recv path** — puppet's original code got this right and
we match it.
`max_polls = 0` means "one non-blocking pass, return whatever's
currently on the ring". Non-zero means spin up to that many
iterations.
**Smoke integration + stress test**
`vmbus.smoke.netvsp_phase3` now, after RNDIS init, does:
1. Send one frame with completion (round-trip).
2. Drain any inbound frames.
3. Send N=128 frames fire-and-forget in a burst, with periodic
non-blocking drains every 16 frames.
4. Final blocking drain.
**Live smoke result on WS 2025 lab machine: 6 passed / 0 failed.**
Output for the stress phase:
netvsp: 1 ethernet frame sent + completion observed
netvsp: drained 0 inbound frames
netvsp stress: 128 frames sent, 0 RingFull hits (auto-recovered)
netvsp stress: final drain got 0 inbound frames
Zero RingFull hits at N=128 confirms the ring absorbs the burst
cleanly (ring = 32 KiB data / ~160 byte packet ≈ 200 packet
capacity, and the host drains continuously). More importantly:
**no hang**. The puppet netvsp hang reproducer with the
`pending_send_sz` protocol fix from bea5f96 in place is
observably stable under sustained load — the classic puppet
symptom (host reader-side transition test with no guest
writer-side feature-bits ⇒ silent deadlock) does not reproduce
here.
To trigger actual RingFull we'd need to either shrink the ring
or send at a rate the host can't keep up with. Neither is
important for the smoke run — the confidence signal is that the
pipeline sustains at least ~130 packets/s without user-visible
progress hitching. If we want a real stress harness later, the
right pattern is a dedicated case that opens a fresh channel
with a smaller (say 2-page) ring so RingFull is guaranteed and
we can assert the retry path works.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
Adds `Netvsp::set_packet_filter(ctx, filter)` — sends an
`RNDIS_SET_MSG` for `OID_GEN_CURRENT_PACKET_FILTER` and waits for
the paired `RNDIS_SET_CMPLT`. **This is required before the host
will deliver any Ethernet frames**: NDIS's default filter is 0,
which silently drops every inbound frame in the netvsp pipeline.
Discovered via a real recv-path test (see follow-on commit for the
smoke case). The chain is:
1. Set MacAddressSpoofing=On + attach test VM to an internal vSwitch
on the host side (host-side config, one-time).
2. Guest sends ARP request for the vSwitch host IP.
3. Host stack replies with ARP `10.99.0.1 is-at <host MAC>`.
4. vSwitch forwards the reply toward our vNIC.
5. WITHOUT set_packet_filter: vSwitch drops the reply with
`Component 43 (our vNIC) DropReason "Packet does not match
destination NIC packet filter"` (observed via
`pktmon filter add -d arp; pktmon start --capture`).
6. WITH set_packet_filter(DIRECTED|BROADCAST|ALL_MULTICAST|
PROMISCUOUS): reply is delivered, guest reads it out of the
transfer-page as a normal RNDIS PACKET_MSG.
Adds:
- Constants: `OID_GEN_CURRENT_PACKET_FILTER = 0x0001010E` and the
standard `NDIS_PACKET_TYPE_*` filter bits (DIRECTED,
MULTICAST, ALL_MULTICAST, BROADCAST, PROMISCUOUS).
- Wire types: `RndisSetRequest` (20 bytes) and `RndisSetComplete`
(8 bytes), field-for-field from openvmm's `rndisprot.rs::SetRequest`
/ `SetComplete`.
- Method: same GPA-direct + xfer-page recv pattern as `rndis_init`,
with the info buffer holding a single u32 for the filter value.
**Follow-on gotcha**: with only DIRECTED|BROADCAST|ALL_MULTICAST,
the vSwitch still dropped the reply because our ARP-request source
MAC (`02:00:00:00:00:01`, locally-administered) doesn't match the
vNIC's assigned MAC (`00:15:5D:...`, Hyper-V-assigned). The reply's
destination MAC is our fake MAC, which doesn't match either filter
condition when only DIRECTED is set. Adding PROMISCUOUS bypasses
that check. A cleaner alternative would be to source the ARP from
the assigned MAC, but that requires the guest to know its own
Hyper-V-assigned MAC (available via an NDIS OID query we haven't
implemented yet).
**Live verification** on WS 2025 lab machine with internal vSwitch
+ host IP 10.99.0.1:
netvsp: RNDIS SET packet_filter=0x2d sent (tid=0x5)
netvsp: RNDIS SET_CMPLT request_id=0x2 status=0x0
netvsp: ARP request microsoft#1 sent for 10.99.0.1
netvsp: inbound after ARP microsoft#1 (42 bytes): [02, 00, 00, 00, 00, 01,
00, 15, 5d, e7, dd, 4a, 08, 06, 00, 01, 08, 00, 06, 04,
00, 02, ...]
netvsp: !!! ARP reply from mac=00:15:5d:e7:dd:4a ip=10.99.0.1
netvsp: drained 1 inbound frames, got_arp_reply=true
Frame decoded:
- dst MAC = 02:00:00:00:00:01 (our sender MAC, correctly targeted)
- src MAC = 00:15:5D:E7:DD:4A (host vEthernet MAC)
- EtherType = 0x0806 (ARP)
- op = 2 (REPLY), sender = 10.99.0.1 @ 00:15:5D:E7:DD:4A,
target = 10.99.0.42 @ 02:00:00:00:00:01
This closes the full VSC data-path bring-up: bidirectional
Ethernet TX + RX proven end-to-end against a real Windows host.
Suite result: 6 passed, 0 failed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
Removes every internal-doc reference (`tasks/vmbus-port-design.md`, `tasks/netvsp-port-design.md`, various `§N.M`s) and replaces each with a self-contained explanation at the point of use. Documentation files themselves are external to this repo and no source file should depend on being read next to them. Touches: - `connection.rs` — inline what the version-negotiation ladder does. - `channel.rs` — explain why monitor-page signalling is out of scope (it's a Copper+ optimisation). - `devices/netvsp.rs` — replace "see the design doc for phases" with a numbered bring-up sequence in the module doc; drop the "phase 3" reference on `rndis_init`. - `gpadl.rs` — inline what the encoder emits (header + N bodies). - `hvsock.rs` — drop the §7 tag; the scope note stands on its own. - `hypercalls.rs` — inline why the ctx owns the input/output page and why guest→host signals always use flag 0. - `interrupt.rs` — reword the "footgun" callout into a forward- progress guarantee that stands without external context. - `lib.rs` — replace the design-doc pointer with a direct link to the two functions this delegates to. - `protocol.rs` — replace "see design doc for citations" with a plain statement, and rewrite the version-ladder doc. - `ring.rs` — inline the "signal only on empty→non-empty" rationale and the pending_send_sz protocol description. - `tests.rs` — drop the §8.1 tags on module comments; explain the wire-size test's reference sources instead. External citation retained: `rndisprot.rs::MESSAGE_TYPE_*` in netvsp.rs still cites MS-RNDIS §2.2 which is a public Microsoft protocol spec (not this repo's internal design doc). 97/97 unit tests pass. UEFI target still builds clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
Address findings from an in-depth code review of vmbus_guest. Ring (ring.rs): * SendRing::write_packet: publish write_idx with SeqCst and reload read_idx after publish for the empty-check, matching openvmm OutgoingRing::commit_write. The pre-publish snapshot could miss a reader park and drop the guest->host wake, hanging TX. * RecvRing::read + drain_signal_decision: SeqCst on both sides to close the Dekker rendezvous with pending_send_sz on aarch64. * write_gpa_direct: reject byte_offset >= PFN region before subtracting (previously underflowed to a huge u64 and silently passed). Interrupt (interrupt.rs): * drain_once: clear the slot then re-read slot[5] & 0x1 to gate EOM (was pre-clear snapshot). Missed EOM could stall SINT2 delivery. * clear_slot: SeqCst fence after the zero, matching Linux vmbus_signal_eom. Hypercalls (hypercalls.rs): * post_message: also retry on InsufficientMemory; raise retries 20 -> 100 with exponential backoff capped at ~10 ms per iter, to match Linux MAX_MSG_RETRY_COUNT / MAX_UDELAY_MS. Netvsp (devices/netvsp.rs): * send_ethernet: track every RNDIS TX buffer in a bounded pending_tx queue and dealloc on the matching VM_PKT_COMP (previously leaked a page per frame in both modes). Cap outstanding TX at 512. * send_ethernet's completion-wait loop now acks any interleaved VM_PKT_DATA_USING_XFER_PAGES with V1_SEND_RNDIS_PKT_COMPLETE. Previously we consumed the packet without acking, starving the host's transfer-page pool and hanging under bidirectional load. * Fire-and-forget send_ethernet opportunistically drains the recv ring before returning so it doesn't fill during sustained bursts. * Scratch buffer 512 -> 4096 to prevent wedging on oversized packets. * rndis_init + set_packet_filter: ack xfer-pages on every skipped branch (wrong set_id, range_count==0, out-of-bounds). * drain_inbound: VM_PKT_COMP now reclaims any matching pending TX. * New helpers: flush_tx (blocks until pending_tx is empty), ack_xfer_page, pending_tx_len. Docs (lib.rs): * Rewrite crate-level rustdoc with a full walkthrough: init, offers, open, ring signal / TX drain discipline, and shutdown. * Add code examples for keyboard bring-up, netvsp bring-up, and a bounded fire-and-forget burst using flush_tx. * Document the single-threaded model and the SeqCst requirement on both sides of pending_send_sz. Tests: 98/98 pass with two new regressions: - drain_once_reads_pending_flag_after_clear - write_gpa_direct_validates_inputs (byte_offset >= region) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
Expand rustdoc across every public module of vmbus_guest with a "When to use this module" section and a compact code example. Complements the crate-level walkthrough added in 2316c52. Modules touched: * connection: private-pump example using negotiate_version + request_offers_with + OfferCollector. * channel: raw channel-over-owned-ring example noting that Channel doesn't own the GPADL or ring memory. * ring: SendRing::write_inband / RecvRing::read snippets and the drain_signal_decision back-pressure protocol; documents the SeqCst requirement on both sides of pending_send_sz. * gpadl: example registering a 16 MiB receive buffer via establish_gpadl. * message: private CompletionTable example, plus a warning that mixing a private table with the default pump loses completions. * interrupt: private SimpPump + poll_until example, linking to MessagePump::poll_until. * synic: init_synic vs deferred (preallocate_synic_pages + init_synic_with_pages) example. * devices/netvsp: full end-to-end example (open -> negotiate -> NDIS config/version -> buffers -> rndis_init -> set_packet_filter -> round-trip send + drain -> fire-and-forget burst + flush_tx), with a gotchas section on drain discipline and the mandatory packet filter. Also removes an unused workspace uefi dependency from Cargo.toml that xtask fmt --fix flagged (allocations already go through alloc::alloc::alloc_zeroed). Validated: cargo check + clippy + doc + test (98/98) + rustfmt all clean on vmbus_guest. No new intra-doc-link or doc-lint warnings introduced; remaining warnings are pre-existing missing docs on keyboard constants and netvsp Version variants. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
Every place in the crate that hands a guest-physical address to the hypervisor was doing an inline 'ptr as u64' cast, relying on the UEFI identity-map invariant to make VA numerically equal to GPA. The assumption is load-bearing across SynIC bring-up, GPADL registration, and GPA-direct RNDIS TX — spread across 6 call sites in synic.rs and devices/netvsp.rs. Consolidate all of these behind a single vmbus_guest::virt_to_phys shim. Today the body is still 'ptr as u64', so the change is a no-op at runtime. But if we ever need to run under a non-identity paging setup (opentmk under its own CR3, paravisor VTL1 layouts, etc.), swapping the body to HvCallTranslateVirtualAddress or a page-table walk in one place picks up every consumer automatically. The shim's rustdoc documents the identity-map invariant (UEFI 2.10 sec. 2.3), why it survives exit_boot_services in our environment, what alternatives exist for non-identity setups, and which addresses to NOT pass through it (host-supplied GPAs, wire-format offsets). Also add a unit test virt_to_phys_is_identity_today that pins the current zero-cost-cast behavior so anyone changing the shim body has to update the test and the docs together. Validated: cargo check + clippy + doc + test (98/98 including the new test) + rustfmt all clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
Four review notes from opentmk/vmbus_guest/notes:
1. ring: SendRing and RecvRing are single-producer / single-consumer
by construction — a ring can't be safely shared across threads
because the pending_send_sz Dekker rendezvous assumes exclusive
writer / reader access to the control page. Add a
PhantomData<*const ()> marker to both so the compiler enforces
!Send + !Sync.
2. ring: The bool returned by write_packet / write_inband /
write_completion / write_gpa_direct is 'need_signal' — true only
when the write crossed the empty->non-empty transition AND the
host hasn't masked interrupts. Callers should signal only when it's
true; signalling on every packet risks Hyper-V's DoS throttling.
Update write_packet's doc and the module-level example, then fix
every caller (keyboard.rs, netvsp.rs — all send_ethernet / RNDIS
TX / xfer-page ack / handshake sites) to honor the bool instead
of discarding it via '_ = need_signal'.
3. Move 'use' statements out of function bodies (synic.rs,
gpadl.rs) up to module scope, and eliminate inline
'use crate::protocol::{...}' in ring_tests. Real function bodies
should be clean; use inside {test} submodules is fine.
4. message: Remove impl Drop for CompletionHandle. The Drop was
redundant with pending()'s retain-and-prune scan, and worse: if a
second register(key) races in between the first handle drop and
the Drop body, the Drop removes the map entry pointing at slot
microsoft#2, orphaning slot microsoft#2's live handle. Drop the impl (and the
now-unused CompletionHandle.table field) — pending() already
handles cleanup lazily, and the Weak in the map means the slot
is unreachable via deliver() the moment the last handle drops.
Also drop a dead 'let ptr = &x as *const u64' in tests.rs (clippy
ref_as_ptr) in favor of core::ptr::from_ref.
Validated: cargo check + clippy + doc + test (98/98) + rustfmt all
clean. No new lints introduced; the reported warnings are all
pre-existing (missing docs on keyboard constants + netvsp Version
variants; result_unit_err on netvsp encode_message / parse_header
helpers; undocumented_unsafe_blocks on OwnedBuf Send/Sync impls).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
Sweep the crate to move fully-qualified core::/alloc::/crate::/ opentmk_core:: references out of function bodies and into use statements at module scope. Cuts through code that read like a UML diagram (e.g. 'crate::hypercalls::post_message(...)') back to plain 'post_message(...)'. Files touched: channel.rs, connection.rs, devices/keyboard.rs, devices/netvsp.rs, gpadl.rs, hvsock.rs, hypercalls.rs, interrupt.rs, message.rs, protocol.rs, ring.rs, synic.rs. Paths that remain qualified are intentional: * devices/netvsp.rs: 'core::result::Result<usize, ()>' on encode_message / parse_header — the crate's own 'Result' alias would shadow it. * error.rs: 'pub type Result<T> = core::result::Result<T, Error>' is the canonical form for a Result alias. Also cleaned up a redundant intra-doc link (`MessagePump::poll_until`) in interrupt.rs. tests.rs still has ~200 fully-qualified references inside '#[cfg(test)] mod xxx' submodules — those are naturally scoped already (each submodule has its own preamble of 'use crate::…') and rewriting them was deferred to keep this commit focused. Validated: cargo check + clippy + doc + test (98/98) + rustfmt all clean. No new warnings introduced. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b2a8c015-cc37-4f2b-a6c6-46c90794de07
Add the guest-side surface for the netvsp fuzzer port:
* vmbus_guest netvsp driver: add public raw-send/renew primitives
- send_nvsp_raw: post a caller-supplied NVSP frame inband
- send_rndis_raw: wrap arbitrary RNDIS bytes in V1_SEND_RNDIS_PKT and
deliver GPA-direct (multi-page PFN list, pending-TX tracking)
- renew_recv_buffer / renew_send_buffer: revoke + re-establish a
buffer reusing the existing GPADL registration
* opentmk_invariant: add functions/netvsp.rs fuzz handlers
(send_nvsp, send_rndis, open_channel, renew_buffer) backed by a
lazily-initialized, kept-warm netvsp session; register them in the
executor and depend on vmbus_guest.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 02421cdc-6ac1-4de3-a547-e68c63a73b12
The netvsp guest fuzz handlers (send_nvsp, send_rndis, open_channel, renew_buffer) previously returned any runtime failure as a handler `Err`, which the syzlang deserializer surfaces as a fatal Error packet that aborts the entire fuzzing campaign. A malformed fuzzing input (a bad guest-memory read, a device-side timeout, or a malformed RNDIS message) is expected noise during fuzzing and must not be campaign-fatal. Wrap the post-parameter-verification body of each handler in a `run_logged` helper that logs the failure via `log::error!` and reports success (`Ok(Void)`), so the fuzz loop advances past the bad testcase. Parameter decoding (`verify_num_params`/`expect_int`) is still validated with `?`, since a bad parameter count/type indicates a fuzzer/grammar bug worth failing on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02421cdc-6ac1-4de3-a547-e68c63a73b12
`HvTestCtx` embeds two inline 4 KiB hypercall pages via `HvCall` (`input_page` + `output_page`), making it ~8 KiB by value. The netvsp fuzz handlers held it as a by-value stack local in `bring_up_session` (and by value in `NetvspSession`), producing an ~8-16 KiB stack frame several calls deep in the testcase-execution path. On the bare-metal UEFI guest there is no stack-growth fault handler, so the frame-entry stack probe touched an unmapped guard page and faulted the guest with a silent #PF (no Rust panic) before `bring_up_session` executed its first statement. This killed the fuzz session on the very first testcase; the fault point shifted with unrelated codegen changes (e.g. added logging), presenting as a Heisenbug. Store the context as `Box<HvTestCtx>` so the ~8 KiB hypercall pages live on the heap and never land on the guest stack. Bring-up now completes and the session survives repeated malformed inputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02421cdc-6ac1-4de3-a547-e68c63a73b12
The fuzzer-facing raw send paths (send_nvsp_raw, send_rndis_raw,
renew_recv_buffer, renew_send_buffer) polled for a host completion using
DEFAULT_MAX_POLLS (~5s). Malformed fuzz input routinely gets no completion,
and a testcase chains several such sends, so a few sends alone exceeded the
fuzzer's 20s TCP read window and tripped a connection timeout ("Failed to
read header", os error 10060), failing the whole loop.
Introduce FUZZ_SEND_MAX_POLLS (~0.25s) for those four fuzz-facing paths so a
timing-out send costs ~0.25s instead of ~5s, keeping a full testcase well
within the fuzzer budget. Bring-up control messages keep DEFAULT_MAX_POLLS
since they complete quickly and must not falsely time out during setup. A
real completion for a well-formed send arrives far faster than the new bound.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 02421cdc-6ac1-4de3-a547-e68c63a73b12
send_nvsp and send_rndis allocated a staging buffer with `vec![0u8; len]` where `len` comes straight from untrusted fuzz input. A garbage length triggered a capacity-overflow panic (alloc/raw_vec) in the guest, leaving it unresponsive until the 30s testcase timeout — which in turn tripped the fuzzer's 20s TCP read window and failed the loop. Reject lengths above MAX_FUZZ_MSG_LEN (64 KiB, matching the driver's MAX_RNDIS_LEN) before allocating, logging the rejection non-fatally via run_logged instead of panicking. Never panic on untrusted guest input. Verified end-to-end: the campaign now runs continuously (1400+ executions, ~9 exec/sec, coverage 656 -> 681) with no panic or TCP disconnect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02421cdc-6ac1-4de3-a547-e68c63a73b12
Sustained fuzzing drove the guest netvsp driver into a permanent `RingFull` state after ~90s: the send path stopped generating any coverage while the campaign kept running. Three coupled issues: 1. pending_tx tracker leak: entries were freed only on a matching host completion, so sends the host silently drops (malformed fuzz input) leaked forever; after PENDING_TX_MAX every send returned RingFull. Fix: reclaim_oldest_tx() force-frees the oldest buffers when the tracker saturates and a drain reaps nothing (safe for a fuzz harness — those buffers were long since consumed by the host). 2. outbound send ring wedge: once our recv ring filled with unreaped completions the host stopped draining our send ring, which then stayed full permanently. The RingFull path returned without draining, so it never recovered. Fix: post_gpa_direct() reaps completions off the recv ring (freeing space without needing send-ring space) and retries the write, breaking the deadlock. 3. shallow ring: the 32 KiB (8-page) outbound ring overflowed on bursty testcases posting many fire-and-forget sends. Enlarge to 128 KiB (32 pages) so typical bursts fit before the host drains. After the fix a live campaign grows coverage and corpus continuously; RingFull is now a transient, self-clearing burst instead of a permanent wedge. vmbus_guest unit tests (98) pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02421cdc-6ac1-4de3-a547-e68c63a73b12
…tapath Certain fuzzer operations (notably renew_buffer on the send buffer) permanently wedge the host netvsp channel: the send buffer is revoked but never re-established, the host stops completing sends, the outbound ring fills, and coverage flatlines (~4 min in, cumulative vmswitch.sys stuck). A consecutive-fault rebuild heuristic proved unreliable because fire-and-forget sends return Ok after merely posting to the ring, resetting the fault counter, so the wedge is never detected. Instead, close and reopen the netvsp channel at every testcase boundary. This makes each testcase self-contained (no wedged ring, revoked buffer, or mutated RNDIS filter can leak across testcases) and deterministically recovers a datapath a prior testcase wedged. close_channel uses SynIC post-messages, not the data ring, so it works even when the ring is full. The VMBus connection (SynIC + offers) is kept up across resets, so a reset costs one channel bring-up, not a full SynIC/request_offers re-init. To keep this leak-free across an unbounded campaign, reclaim the guest memory the old channel owned. OwnedBuf, the ring region, and pending-TX staging buffers have no Drop, so into_channel() leaked ~17 MiB per teardown and OOM'd the guest after ~40 testcases. Add Netvsp::into_parts + NetvspBacking::free to deallocate the ring region, the recv/send GPADL buffers, and outstanding TX buffers after the channel is closed. Finally, shrink the recv buffer from 16 MiB to 256 KiB. Its GPADL is re-registered on every reset, and a 16 MiB buffer (4096 PFNs) streams as ~140 GPADL_BODY post-messages, halving exec/sec. This fuzzer barely exercises RX, so 256 KiB (~3 messages) is ample and restores throughput to the pre-reset baseline with no loss of host-side coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02421cdc-6ac1-4de3-a547-e68c63a73b12
…nd-timeout budget
Two throughput fixes for the netvsp live fuzzer, backed by ms-precision
per-testcase profiling.
1. Reset was falling back to a full VMBus unload + SynIC re-init +
RequestOffers + reopen on *every* testcase (100% of cheap reopens
failed). Root cause: reset_session closed the channel and freed the
ring/recv/send backing, but never tore down their GPADLs. The next
alloc_zeroed hands the just-freed guest pages straight back, so the
reopen re-registered a fresh GPADL over the exact PFNs the host still
held under the old handle and the host NAK'd the GpadlHeader
(Parse{ty:GPADL_CREATED,"non-success"}).
Fix: tear down the ring + recv + send GPADLs on reset before freeing
the backing. The teardown must happen while the channel id is still
live host-side, so split close into a keep-relid variant
(close_channel_keep_relid) that posts CloseChannel WITHOUT
RelIdReleased -- matching vmbus_client, which sends RelIdReleased only
after teardown. Releasing the relid first makes the host drop the
GpadlTeardown and the guest times out. Retaining the relid also lets
the reopen skip RequestOffers.
Result: cheap channel-only reopen now succeeds 100% of the time; no
full rebuilds.
2. Lower FUZZ_SEND_MAX_POLLS 5M -> 2M (~0.25s -> ~0.1s). Live
measurement: genuine completions arrive p50 ~23ms / 97% within ~75ms,
while ~53% of fuzz sends get no completion and spin the whole budget.
~0.1s keeps effectively all real acks with margin and halves the
wasted spin. A missed late ack is harmless -- the packet was already
delivered via post_message and the per-testcase reset discards queued
completions.
Combined: ~0.60 -> ~1.02 exec/sec (~70%) on the network-backed datapath;
send-op p50 249ms -> 104ms. 98/98 vmbus_guest unit tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 02421cdc-6ac1-4de3-a547-e68c63a73b12
The guest logger writes every record byte-by-byte over COM2, spinning on the UART transmit-holding register until each byte drains (serial.rs). Each ~150-250B JSON-wrapped line stalls guest execution, and the netvsp reopen + RNDIS send path emitted ~10-20 such lines per testcase, which dominated per-testcase time. - Raise the serial max level from Debug to Info so debug/trace records short-circuit inside the log! macros (no format, no lock, no serial). - Demote the 19 hot-path netvsp per-op info logs to debug. Diagnostics still available by flipping the filter back to Debug. warn/ error and one-time init logs are unchanged. Measured: per-testcase cadence 966ms -> 194ms p50; exec/sec 1.02 -> ~4.7. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02421cdc-6ac1-4de3-a547-e68c63a73b12
Port the legacy puppet VMBus fuzzer's guest side into opentmk.
vmbus_guest (raw escape hatches for the fuzzer):
- fuzz.rs: resolve_connection_id, post_raw_message{,_wait}, drain_pending,
relids, and RawChannel (open any offer, raw ring writes, drain, close),
plus a UEFI-gated poll_capture. Truncates/logs rather than panicking on
untrusted input.
- ring.rs: SendRing::write_raw_packet writes a caller-supplied 16-byte
descriptor verbatim (length8/data_offset8/flags may be inconsistent with
the real ring contents) while guest bookkeeping advances by real bytes.
Extracted reserve/commit helpers.
- interrupt.rs: drain_once_capture returns raw reply bytes and only logs
route_message failures, so a fuzzed reply can't wedge the SINT2 pipe.
slot_offset is now pub const fn.
opentmk_invariant:
- functions/vmbus.rs: vmbus_msg, vmbus_msg_comp, vmbus_packet,
vmbus_reopen_channel, vmbus_fill_relids handlers plus reset_session,
backed by a lazily brought-up cached VmbusSession.
- executor: register the five handlers and call vmbus::reset_session()
per testcase.
- netvsp.rs: formatting only (cargo xtask fmt --fix).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1d841cee-18a0-4d14-ad43-07955ed0ac80
Add debug-gated logging to the VMBus fuzzer's Layer 2 handlers and the raw-channel open path to aid crash triage. All logs are at debug level so normal (Info) campaign runs stay quiet and fast over the byte-at-a-time COM2 console, matching the existing debug-trace style in interrupt.rs and message.rs. Flip the log level to Debug (tmk_logger.rs) to replay a saved crash input and see which handler / open step wedged the guest. - vmbus.rs: per-handler `enter` logs (vmbus_msg, vmbus_msg_comp, vmbus_packet, vmbus_reopen_channel, vmbus_fill_relids), the raw-send result in vmbus_packet, and the channel-open offer selection. - fuzz.rs: RawChannel::open step logs (alloc, gpadl established, open_channel done). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1d841cee-18a0-4d14-ad43-07955ed0ac80
- post_raw_message() in vmbus_guest/fuzz.rs now silently drops messages with client-only MessageType values (1,2,4,6,10,12,15,17,20,23,24,26,28) before posting to host, preventing host vmwp.exe crash from Rust unreachable! panic in Vmbusr.dll channels.rs:3729. - reset_session() in opentmk_invariant rewritten with full per-testcase connection cycle (drain/close/Unload/InitiateContact/RequestOffers) and debug step markers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1d841cee-18a0-4d14-ad43-07955ed0ac80
|
This PR modifies files containing For more on why we check whole files, instead of just diffs, check out the Rustonomicon |
|
That is a lot of code. Is there no way we can reuse our existing vmbus infrastructure? |
|
let me pull up the analysis from a few weeks back. |
There was a problem hiding this comment.
🟡 Changes recommended
Critical lifecycle and memory-safety issues, along with VMBus session and protocol handling defects, remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds a no_std guest-side VMBus client and integrates VMBus/NetVSP fuzzing into opentmk_executor.
Changes:
- Implements SynIC, protocol messaging, channels, GPADLs, interrupts, and hypercalls.
- Adds keyboard support and raw VMBus fuzzing primitives.
- Registers handlers and updates workspace dependencies and logging.
File summaries
| File | Description |
|---|---|
opentmk/vmbus_guest/src/synic.rs |
SynIC setup and page allocation |
opentmk/vmbus_guest/src/protocol.rs |
VMBus wire types |
opentmk/vmbus_guest/src/message.rs |
Message encoding and completion routing |
opentmk/vmbus_guest/src/lib.rs |
Public client API and documentation |
opentmk/vmbus_guest/src/interrupt.rs |
SIMP polling and message handling |
opentmk/vmbus_guest/src/hypercalls.rs |
Hypercall wrappers |
opentmk/vmbus_guest/src/hvsock.rs |
Hvsock helpers |
opentmk/vmbus_guest/src/gpadl.rs |
GPADL lifecycle |
opentmk/vmbus_guest/src/fuzz.rs |
Raw VMBus fuzzing primitives |
opentmk/vmbus_guest/src/error.rs |
Error definitions |
opentmk/vmbus_guest/src/devices/mod.rs |
Device module exports |
opentmk/vmbus_guest/src/devices/keyboard.rs |
Synthetic keyboard driver |
opentmk/vmbus_guest/src/connection.rs |
Version negotiation and connection state |
opentmk/vmbus_guest/src/channel.rs |
Channel lifecycle |
opentmk/vmbus_guest/Cargo.toml |
Guest crate dependencies |
opentmk/opentmk_executor/src/functions/vmbus.rs |
VMBus fuzz handlers |
opentmk/opentmk_executor/src/functions/netvsp.rs |
NetVSP fuzz handlers |
opentmk/opentmk_executor/src/functions/mod.rs |
Function module registration |
opentmk/opentmk_executor/src/executor/mod.rs |
Executor integration and reset flow |
opentmk/opentmk_executor/Cargo.toml |
Executor dependencies |
opentmk/opentmk_core/src/tmk_logger.rs |
Logging level adjustment |
Cargo.toml |
Workspace membership and dependencies |
Cargo.lock |
Dependency lock updates |
Review details
Suppressed comments (13)
opentmk/opentmk_executor/src/functions/vmbus.rs:300
conn_id == 0is resolved beforewith_sessioninitializes the VMBus connection. On first use this permanently chooses fallback ID 4, so any host-selected connection ID is ignored even thoughresolve_connection_idis documented to use the negotiated value. Move the resolution into thewith_sessionclosure after bring-up.
let connection_id = fuzz::resolve_connection_id(conn_id);
opentmk/opentmk_executor/src/functions/vmbus.rs:334
- This completion variant has the same first-use ordering bug:
conn_id == 0is resolved beforewith_sessionperforms negotiation, so the negotiated connection ID can be ignored. Resolve the ID inside the closure after the session exists.
let connection_id = fuzz::resolve_connection_id(conn_id);
opentmk/vmbus_guest/src/connection.rs:343
- On timeout the old
VersionResponseis still allowed to arrive, but the next loop registers the same singletonCompletionKey::VersionResponse. A delayed response from the previous request is then delivered to the new handle and can make the guest build state for the wrong version. Do not fall through on timeout without fencing or draining the old response, or restart the connection.
Err(Error::Timeout) => {
log::warn!("negotiate: timeout on version {version:?}, trying next in ladder");
continue;
opentmk/vmbus_guest/src/devices/keyboard.rs:295
write_and_signalunconditionally requests a VM_PKT_COMP, butset_ledsreturns immediately and never drains that completion. Repeated LED updates will accumulate completion packets in the receive ring until it fills and blocks the channel; either make LED writes fire-and-forget or consume their completions.
let mut flags = PacketFlags::new();
flags.set_request_completion(true);
let need_signal = self.send.write_inband(payload, flags, 0)?;
opentmk/vmbus_guest/src/error.rs:48
- This error discards the underlying
TmkError, so callers only seehypercall failedand cannot distinguish access denial, invalid input, or resource exhaustion throughDisplay. Include the source status in the error message to make failures actionable.
#[error("hypercall failed")]
Hypercall(#[from] TmkError),
opentmk/vmbus_guest/src/hypercalls.rs:42
- This API rejects oversized payloads at lines 56-60; it does not truncate them. The doc comment therefore promises behavior callers will not get and should say that payloads above
HV_MESSAGE_PAYLOAD_SIZEreturnError::Parse.
/// The payload is truncated at `HV_MESSAGE_PAYLOAD_SIZE` (240 bytes).
opentmk/vmbus_guest/src/interrupt.rs:376
- The SIMP page is written asynchronously by the hypervisor, but this path exposes it as a normal
&mut [u8]and uses ordinary byte loads/stores (slot[0],slot[5], and payload copies). Those accesses are not a shared-memory synchronization primitive and can observe stale or torn data on weakly ordered targets; the existing VMBus ring and SynIC emulator use atomic shared-memory operations for this purpose. Read, copy, and clear the slot through atomic or otherwise explicitly shared-memory-safe operations.
let first_byte = slot[0];
opentmk/vmbus_guest/src/lib.rs:85
OfferChannelexposesinstance_id, notinterface_instance, so this public example cannot be copied as written.
//! offer.interface_instance,
opentmk/vmbus_guest/src/lib.rs:114
- This public keyboard example calls
Keyboard::open, but the driver exposesKeyboard::newand requires the caller to construct the ring views. The example therefore refers to an API that does not exist.
//! let mut kbd = keyboard::Keyboard::open(&mut ctx, kbd_offer)?;
opentmk/vmbus_guest/src/lib.rs:113
Error::NotFoundis not defined by this crate'sErrorenum, so the public example cannot compile when copied. Use an existing error or add a deliberate missing-offer error variant.
//! .ok_or(vmbus_guest::Error::NotFound)?;
opentmk/vmbus_guest/src/lib.rs:135
Error::NotFoundis also used here, but that variant does not exist in this crate's error surface, so the netvsp example has the same copy-and-compile failure.
//! .ok_or(vmbus_guest::Error::NotFound)?;
opentmk/vmbus_guest/src/message.rs:286
- The process-wide table is used by every high-level UEFI path, and GPADL/open requests use new monotonically increasing keys. Once a handle is dropped, this
Weakentry is never pruned unless a caller explicitly invokespending()(the high-level paths do not), so a long fuzz campaign grows theBTreeMapwithout bound. Prune dead entries while registering, or make cleanup automatic.
self.inner.entries.lock().insert(key, Arc::downgrade(&slot));
opentmk/vmbus_guest/src/synic.rs:196
- This readback is only logged; none of the four returned values is compared with the values just written. Consequently
program_synic_registersstill returnsOk(())if the hypervisor silently changes or rejects a register, contrary to the verification described above. Compare the expected register values and return an error on mismatch.
let readback = get_vp_registers(
ctx,
HvInputVtl::CURRENT_VTL,
&[
HvRegisterName(HV_REGISTER_SIMP),
- Files reviewed: 24/26 changed files
- Comments generated: 11
- Review effort level: Lite
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if let Err(e) = vmbus_guest::gpadl::teardown_gpadl(&mut *ctx, ring_gpadl) { | ||
| log::warn!( | ||
| "netvsp: reset teardown ring gpadl {:?} failed: {e:?}", | ||
| ring_gpadl.id() | ||
| ); |
| if let Err(te) = teardown_gpadl(ctx, gpadl) { | ||
| log::warn!("raw channel: gpadl teardown after failed open: {te:?}"); |
| } = self; | ||
| let ring_gpadl = channel.ring_gpadl(); | ||
|
|
||
| let close_res = close_channel(ctx, channel); |
| if let Some(handler) = *HANDLER.lock() { | ||
| handler(result); | ||
| } | ||
| } |
| netvsp::reset_session(); | ||
| vmbus::reset_session(); |
| let total_len_bytes: u16 = range_payload | ||
| .len() | ||
| .try_into() | ||
| .expect("GPADL range payload exceeds u16 length"); |
| if let Err(e) = route_message(&payload_buf[..payload_len], table, sink) { | ||
| if !tolerant { | ||
| return Err(e); |
| handle: &CompletionHandle, | ||
| sink: &mut dyn MessageSink, | ||
| ) -> Result<()> { | ||
| let table = completion_table(); |
| /// Allocate SIMP + SIEFP pages via `uefi::boot::allocate_pages` | ||
| /// without programming the registers. Useful for callers that want | ||
| /// to control when `exit_boot_services` happens relative to the | ||
| /// hypercalls.#[cfg(target_os = "uefi")] |
| //! * `InitiateContact`/`VersionResponse` negotiation with newest-to-oldest | ||
| //! fallback down to `Version::Copper` (6.0). |
|
I think the biggest issue is that vmbus client is std and we need no_std |
|
I would much rather we investigate what would be required to make our existing vmbus code no_std instead of duplicating all of it. That'd also provide us fuzzing coverage benefits, as we'd cover the same implementation used in OpenHCL |
| // at the `max_level()` check, so they are never formatted, never lock the | ||
| // writer, and never stall the guest spinning on the UART transmit register. | ||
| // Bump this to `Debug` when diagnosing to re-enable the verbose per-op logs. | ||
| log::set_logger(&LOGGER).map(|()| log::set_max_level(log::LevelFilter::Info)) |
There was a problem hiding this comment.
We should leave this on debug for test scenarios
|
Or alternatively, we wait to take this work until rust-lang/rust#100499 is available. |
This adds vmbus + netvsc guest client code so that opentmk_executor is able to fuzz netvsp/vmbus