Windows: add gated direct I/O for mounted raw volumes

[0x1337F00D]

Summary

This adds a second, opt-in I/O path for mounted raw partitions and drives on Windows. The existing encrypted I/O queue remains the compatibility path; eligible raw-volume reads and writes can independently use a bounded direct path when their respective gates are enabled.

The implementation was developed and measured only in a Windows 11 ARM64 Parallels VM with a locally test-signed driver. It has not yet been validated on bare-metal NVMe hardware or on x64 Windows.

Related performance issue: #136.

Design

• Two independent, default-off registry gates:
  • VeraCryptEnableMountedVolumeFastReadIo
  • VeraCryptEnableMountedVolumeFastWriteIo
• Only mounted raw partitions/drives are eligible.
• File containers, system-encryption/filter I/O and unsupported requests keep the legacy queue.
• A separate asynchronous raw-device handle prevents direct requests from changing the synchronous semantics expected by the legacy path.
• Each mounted volume owns a bounded pool of 8 workers with at most 8 direct operations in flight.
• Requests up to 1 MiB are accepted and segmented according to the host device's actual maximum transfer and alignment constraints.
• Overlapping writes are range-ordered. Reads and writes cross explicit barriers where required, preserving visibility and flush/dismount ordering.
• Compatibility or eligibility failures fall back to the existing queue.
• Stop/dismount closes admission, drains queue-owned work, and retains remove-lock/outstanding ownership until each child IRP actually completes.

The lifetime model intentionally does not abandon a timed-out child IRP. This addresses the ownership problem discussed in prior direct-I/O work such as #1808.

Performance-counter IOCTLs used to prove path selection during development are intentionally excluded from this product PR; they remain isolated for a possible separate diagnostics PR.

Performance evidence

DiskSpd was run against the same 4 GiB file on a mounted raw test partition (W:), with the same profile order and arguments for legacy (gates off) and direct modes:

• Sequential: -d8 -W0 -C0 -b1M -o4 -t2 -Sh -L
• Random: -d8 -W0 -C0 -b4K -o32 -t4 -r -Sh -L
• Reads used -w0; writes used -w100

Profile	Legacy path	Direct path	Change
Sequential read	3232.81 MiB/s	5598.72 MiB/s	+73.18% (1.7318x)
Random read, 4 KiB	17,403.70 IOPS	37,960.88 IOPS	+118.12% (2.1812x)
Sequential write	1386.84 MiB/s	2277.34 MiB/s	+64.21% (1.6421x)
Random write, 4 KiB	6,104.56 IOPS	12,680.78 IOPS	+107.73% (2.0773x)

These figures are VM/cache/hypervisor-sensitive and must not be read as bare-metal NVMe claims. They demonstrate the relative bottleneck reduction in this test environment.

Counters in an instrumented sibling build confirmed that eligible bytes traversed the direct path. The final PR tree removes that counter ABI but retains the same direct-I/O data path plus the final lifetime/ordering hardening.

Stability and correctness checks

The evidence run completed with:

• 3 encrypted-system reboots without recovery
• 3 mount/dismount cycles
• chkdsk exit code 0
• unchanged file hash/marker across the matrix
• direct-path counter deltas for eligible profiles and zero idle deltas
• no re-encryption and no new VM snapshot

The final counter-free ARM64 Release driver also builds successfully (veracrypt.sys SHA-256 B1B625604BC8FDFF021F53260EEAC37C5D30E1F9D0FDC6D81C24BBE8915BCBB8).

Scope and requested review

This PR does not change the system/filter path, file-container path, or Windows storage identity/PnP behavior. It is specifically the dual-path performance change for already mounted raw volumes.

Review and measurements on bare-metal Windows, especially x64 and fast NVMe devices, would be very valuable before enabling either gate by default.

[idrassi]

Thanks for including measurements and being clear about the VM setup. I can see why this is worth exploring, but I'm requesting changes before we take it into master. I have kept the specific code concerns inline.

The next measurements need to cover mixed traffic. A pending write blocks new direct reads across the volume, even when the ranges don't overlap. Once eight direct reads have been admitted, further reads take the legacy path. The current pure read and pure write profiles don't tell us what happens as requests move between those paths.

After the corrections, I would prefer a smaller proposal that accelerates reads first. Concurrent direct writes deserve a separate review once the earlier work has been tested under mixed traffic.

[idrassi]

I'm concerned about calling TCCompleteDiskIrp directly from these workers. An upper completion routine can submit another eligible read for the same volume and wait for it. If all eight workers do that, their new requests are queued with nobody left to process them. Releasing read admission before the callback allows those requests in: it doesn't free a worker.

I think we should keep original request completion separate from the workers needed to service further host I/O. Reentrancy during completion is why we introduced deferred completion in 42fdbcf and extended it to writes in 453ff28. Adding more workers doesn't remove the cycle.

[idrassi]

ThreadBlockReadWrite is checked before entering the loop, but only Cancel is checked between segments. InvalidateVolumeKeys can set the block and erase the key schedules while this request is still running.

In a test, I split a 128 KiB read into two 64 KiB segments and triggered key erasure after the first. The second read and decrypt still ran, and the task reported success for the full request. IoThreadProc checks the block before each legacy fragment: here we're starting further operations after it has already been set.

DirectHostWriteTask also continues issuing segments after the block changes, although its buffer has already been encrypted. Please stop new segments once the block is set and account for key erasure racing the decryption itself. Another flag check alone won't cover that latter race.

[idrassi]

This fails Release compilation because remaining is used only by ASSERT. Once the assertion is compiled out, MSVC reports C4189, which becomes C2220 with warnings treated as errors.

I ran the Driver.vcxproj ClCompile target with MSVC 14.44.35207 and WDK 10.0.26100.0. Both x64 Release and ARM64 Release failed here, x64 Debug passed. Please fix the unused variable without relaxing the warning policy.

I would also like the exact command used for the reported ARM64 Release build, since I can't reproduce that result.

[idrassi]

The PR description calls these VeraCryptEnableMountedVolumeFastReadIo and VeraCryptEnableMountedVolumeFastWriteIo, but the driver reads names without Enable.

Please make the names agree.

[idrassi]

Setting the write option to zero and reloading the configuration doesn't stop direct writes on a volume that already has them enabled. CanUseDirectHostWrite checks queue->DirectHostWriteEnabled, which was set when the volume was mounted, rather than this global. Read dispatch does consult the reloaded global, so the two options behave differently.

Are these intended to take effect only when a volume is mounted? That's a reasonable choice, but it needs to be consistent and documented. If changes are meant to apply immediately, the write path needs a proper transition that accounts for requests already in progress.

[idrassi]

DirectHostWriteConfigured can be true for a writable hidden volume even though CanUseDirectHostWrite rejects every write to it. The same applies to an outer volume mounted with hidden volume protection, provided the second host handle opens.

This condition then enables ordered flush handling for those volumes even when OrderedFlushBarriersEnabled is false. We put that behavior behind its own option after compatibility problems, so I don't want it enabled implicitly for volumes that can't use the new write path.

Please preserve the existing flush choice for those excluded volumes. Volumes that have actually used direct writes still need their outstanding work accounted for before any change in flush handling.

[idrassi]

Please check DataLength == sizeof(uint32) before reading data->Data, here and in the write option below. REG_DWORD tells us the type, but it doesn't guarantee that the stored value contains four bytes.

TCReadRegistryKey allocates the size returned by the registry query, so a short value can make this read beyond the payload and possibly beyond the allocation. The main configuration flags decoder already checks the length. These new options should stay disabled when their values are malformed.

[idrassi]

We can't inspect irp after TCCompleteDiskIrp. IsWriteBarrierIrp dereferences its current stack location, but completion may already have released the IRP or changed those stack locations.

Please capture whether this is a write barrier before completing it, then use the saved value for the accounting. Normal item preallocation makes this failure branch unusual, but it still needs to be valid when reached.

[idrassi]

Were the benchmark volumes mounted with bExclusiveAccess set? The earlier open of hDeviceFile uses share mode zero for an exclusive writable mount. On a stack that enforces those sharing checks, this second open conflicts with the first and the direct path stays unavailable.

Please verify this on an actual raw device with both exclusive and shared mounts, and show that the direct path was selected in each supported case. I don't want us to weaken the original exclusive open just to make the second handle work.

[idrassi]

The existing IoThreadProc impersonates SecurityClientContext before issuing host requests. These workers don't, including when they fall back to ZwReadFile or ZwWriteFile on the legacy handle.

Is using the system thread's security context intentional for this path? The handle was opened through the existing mount code, so this difference alone doesn't establish an access bypass. It does change the context seen by lower filters, though. Please explain why it's safe here and include it in the compatibility testing.