Skip to content

Allow retrieving compressed chunks #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
petrutlucian94 merged 1 commit into from
Sep 17, 2026
Merged

Allow retrieving compressed chunks #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Overview

- This is a test project meant to bypass/replace VDDK, which is no longer
- This is a project meant to bypass/replace VDDK, which is no longer
publicly available.
- The end goal is to have a Python library that can be used as a VDDK replacement
to retrieve VMware disk contents.
Expand Down
9 changes: 7 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ OpenVixDiskLib is an open-source Python replacement for VMware VDDK's
`VixDiskLib` NBD path. It reads and writes VMDK contents over vSphere
NFC without the proprietary VDDK SDK.

AI tools (Cursor + Grok 4.6) have been heavily used to reverse engineer the
NBD and NFC protocols, obtaining a working VDDK replacement in a few hours and
comprehensive testing in a matter of days.

The Python package is `openvixdisklib` (lowercase, following usual
Python naming).

Expand All @@ -18,7 +22,7 @@ Implemented against vCenter 8 / ESXi 8. Default transport is `nbdssl`

- `VixDiskLib_ConnectEx` (UID credentials)
- `VixDiskLib_Open` (datastore path, read-only or read-write)
- `VixDiskLib_Read`
- `VixDiskLib_Read` (optional ``skip_decompression`` packs FastLZ extras)
- `VixDiskLib_Write`

Not implemented: compression open flags other than FastLZ, CBT /
Expand Down Expand Up @@ -112,7 +116,8 @@ tox -e integration -- --runslow

Compare write/read throughput of OpenVixDiskLib and native VDDK
(`64KiB`, 129-sector, and `32MiB` transfers; `nbdssl` and `nbd`;
plain and FastLZ):
plain, FastLZ, and OpenVixDiskLib FastLZ ``skip_decompression``;
AIO sessions 64 KiB×1, 1 MiB×1, 2 MiB×1, and 2 MiB×4).

```bash
tox -e perf
Expand Down
39 changes: 29 additions & 10 deletions docs/nfc_open.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,8 +158,26 @@ obtain a file handle or to read sector 0.

### OPEN_SESSION / sockopts / resource pool

VDDK sends 16 zero bytes (`OPEN_SESSION`; server replies with 16 zeros),
12 zero bytes (`SET_SOCK_OPTS`; server returns send/recv buffer sizes
`OPEN_SESSION` payload is 16 bytes, little-endian:

| Offset | Type | Meaning |
| ------ | -------- | ---------------------------------------------------- |
| 0 | `uint32` | 0 (unused in captures) |
| 4 | `uint32` | AIO buffer size in **bytes** (VDDK default 65536) |
| 8 | `uint32` | Buffer count (VDDK ``nfcAio.Session.BufCount``) |
| 12 | `uint32` | 0 |

VDDK config `vixDiskLib.nfcAio.Session.BufSizeIn64KB` is that byte size
divided by 64 KiB (`1` → 65536, `32` → 2097152). The server replies
with 16 zeros; it still **uses** the requested size for IO extras.
A 129-sector read is two fragments at 64 KiB, and one 66048-byte
fragment at 2 MiB. Lab ESXi 8 accepted 2 MiB (`BufCount` 1 and 4) and
rejected 16 MiB and 32 MiB (`OPEN_SESSION` AIO error). Broadcom's 16 MiB
figure is session memory (`size × count`), not a larger extra; the
per-buffer max on the wire is 2 MiB. Probe:
`docs/probing_samples/vddk_aio_bufsize_probe.py`.

`SET_SOCK_OPTS` is 12 zero bytes (server returns send/recv buffer sizes
and a `uint32` flag), then `uint32` 1 (`SET_RES_POOL`, log: “Setting
Resource Pool(1)”).

Expand Down Expand Up @@ -205,14 +223,15 @@ classic type 4 `NFC_SESSION_COMPLETE`.

## OpenVixDiskLib

| Piece | Module |
| ----------------------------- | ----------------------------------------------- |
| VIM + authd | `openvixdisklib.nfc_auth.authenticate` |
| Dup fd, skip TLS for NFC | `openvixdisklib.nfc_open.takeover_authd_socket` |
| Second TLS for nbdssl | `openvixdisklib.nfc_open.wrap_nfcssl_socket` |
| FastLZ for NBD compression | `openvixdisklib.fastlz` (pip `pyfastlz`) |
| Handshake + AIO + OPEN_FILE | `openvixdisklib.nfc_open.open_disk` |
| Sector read / write / close | `openvixdisklib.nfc_open.NfcDisk` |
| Piece | Module |
| ------------------------------- | --------------------------------------------------------- |
| VIM + authd | `openvixdisklib.nfc_auth.authenticate` |
| Dup fd, skip TLS for NFC | `openvixdisklib.nfc_open.takeover_authd_socket` |
| Second TLS for nbdssl | `openvixdisklib.nfc_open.wrap_nfcssl_socket` |
| FastLZ for NBD compression | `openvixdisklib.fastlz` (pip `pyfastlz`) |
| Handshake + AIO + OPEN_FILE | `openvixdisklib.nfc_open.open_disk` |
| AIO extra size / pool count | `open_disk(..., aio_buffer_size=, aio_buffer_count=)` |
| Sector read / write / close | `openvixdisklib.nfc_open.NfcDisk` |

Run:

Expand Down
86 changes: 70 additions & 16 deletions docs/nfc_read.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,13 @@ length = numSectors * sectorSize
| `VixDiskLib_Read(h, 0, 128, buf)` | IO length 65536 (AIO buffer size), one fragment |
| `VixDiskLib_Read(h, 0, 129, buf)` | One request of 66048; **two** reply fragments |

VDDK does **not** split a `Read` larger than 64 KiB into multiple
requests. The client sends one AIO message; the server answers with
one or more same-`opId` replies, each carrying at most
`NFC_AIO_BUFFER_SIZE` (65536) data bytes. `NfcAioInitSession` logged
that buffer size and count 4 during open.
VDDK does **not** split a `Read` larger than the AIO buffer into
multiple requests. The client sends one AIO message; the server
answers with one or more same-`opId` replies, each carrying at most
the OPEN_SESSION buffer size (VDDK default 65536).
`vixDiskLib.nfcAio.Session.BufSizeIn64KB=32` advertises 2 MiB; a
129-sector read then returns **one** 66048-byte extra, and a 2 MiB +
512 read returns 2097152 + 512. See `docs/nfc_open.md` (OPEN_SESSION).

Sparse regions are still transferred as zeros. A read of 8 sectors at
LBA 8 on this disk was 4096 zero bytes on the wire, not a skip.
Expand Down Expand Up @@ -81,22 +83,23 @@ payload + `chunkLength` data bytes.

Reply payload (handle is zeroed; lengths describe this fragment):

| Offset | Type | Meaning |
| ------ | -------- | ----------------------------------------------- |
| 0 | `uint64` | `0` |
| 8 | `uint64` | `1` (read) |
| 16 | `uint64` | Byte offset of the **request** |
| 24 | `uint32` | Total request length |
| 28 | `uint32` | Byte offset of this fragment (`0`, `65536`, …) |
| 32 | `uint32` | This fragment’s byte length |
| 36 | `uint32` | Same as offset 32 |
| 40 | `uint32` | `0` |
| Offset | Type | Meaning |
| ------ | -------- | -------------------------------------------------------------------- |
| 0 | `uint64` | `0` |
| 8 | `uint64` | `1` (read) |
| 16 | `uint64` | Byte offset of the **request** on disk |
| 24 | `uint32` | Total request length |
| 28 | `uint32` | Fragment byte offset **in this request** (`0`, `65536`, …), not disk |
| 32 | `uint32` | This fragment’s uncompressed byte length |
| 36 | `uint32` | Same as offset 32, or compressed extra size when type is FastLZ |
| 40 | `uint32` | `0` |

When there is a single fragment, offsets 24–31 look like a `uint64`
length (the fragment offset is 0). The 129-sector capture shows why
they are two `uint32`s: fragment 0 has `(66048, 0)` then chunk 65536;
fragment 1 has `(66048, 65536)` then chunk 512. `0x00010000` at offset
28 is the byte offset, not a 0-based index.
28 is the byte offset, not a 0-based index. Disk byte address of a
fragment is request offset (payload 16) plus payload 28.

Read loop: receive fragments with that `opId` until the concatenated
data length equals the request. Use the `uint32` at payload offset 32
Expand All @@ -113,6 +116,9 @@ S: type=7 opId=18 size=44 dest=0 chunk=65536 + 65536 data
S: type=7 opId=18 size=44 dest=65536 chunk=512 + 512 data
```

`dest` in that dump is payload offset 28 (`ReadFragment.dest`): 0 and
65536 are positions in this 66048-byte read, not sector numbers.

## Lab check

Integration tests create an empty 10 GiB thin disk, write a repeating
Expand All @@ -135,6 +141,54 @@ The integration test writes and then reads the captured VDDK ranges
(including a 129-sector transfer that must assemble two read
fragments).

## Skip decompression (OpenVixDiskLib extension)

`VixDiskLib_Read` always fills `buf` with uncompressed sector bytes.
OpenVixDiskLib can skip FastLZ decode so a backup application can
forward the compressed data as-is, avoiding unnecessary re-compression.

`NfcDisk.readinto(..., skip_decompression=True)` and
`VixDiskLibHandle.read(..., skip_decompression=True)` still send one
IO request and wait until uncompressed `filled == length`. They do
**not** decompress. Extras are packed densely from offset 0 of `buf`.
`ReadResult.fragments` describes each extra. Type `2` extras are
FastLZ; type `0` fallbacks are raw. Concatenating extras is not a
valid FastLZ stream; the caller must use the table to split them.

| Field | Meaning |
| ---------------------- | ------------------------------------------------------------------------------------------------ |
| `dest` | Byte offset **in this uncompressed read** (NFC payload 28). Not a disk LBA or VMDK file offset. |
| `uncompressed_length` | Uncompressed fragment size (NFC payload 32). |
| `compression_type` | `NFC_COMPRESSION_NONE` (0) or `NFC_COMPRESSION_FASTLZ` (2). |
| `offset` | Start of this extra in packed `buf` (receive order, densely from 0). |
| `length` | Extra size on the wire. |

Disk byte address of a fragment is `start_sector * 512 + dest`. A
129-sector `read` from sector 0 or from sector 1000 still reports
`dest=0` and `dest=65536` when extras are 64 KiB.

`buf` is sized for the uncompressed request, so it is always large
enough. Default `read` still decompresses; `fragments` is empty and
`compressed_length` is still the extra bytes on the wire.
`skip_decompression` with a plain (no FASTLZ) open only records raw
extras (`compressed_length == uncompressed_length`).

This is not `VixDiskLib_Read`. Do not add an open flag for it;
compression on the wire is already the FASTLZ open flag.

A 32 MiB read at 64 KiB extras is 512 fragments in **one** result. A
2 MiB OPEN_SESSION extra (`aio_buffer_size=2097152`) is 16 fragments
for the same read. One dest PUT per extra is not viable.

```
uncompressed request (offsets in this read, not on disk)
|---------------- 64KiB --|-- 64KiB --|-- ... --|
dest=0 dest=65536
extra (FastLZ or raw) extra (FastLZ or raw)

buf when skip_decompression=True: extras packed densely from offset 0
```

## What is still VDDK-only

- zlib and skipz NBD compression flags
Expand Down
11 changes: 6 additions & 5 deletions docs/nfc_write.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,9 @@ A 1-sector VDDK write was 572 bytes on the wire: 16 + 44 + 512.

## Fragments and the single reply

`NfcAioInitSession` advertises a 64 KiB buffer. Extra per type-7
message is at most that size. VDDK does **not** issue a new `opId` per
`OPEN_SESSION` advertises the AIO buffer size (VDDK default 64 KiB;
`BufSizeIn64KB` can raise it). Extra per type-7 message is at most
that size. VDDK does **not** issue a new `opId` per
chunk, and it does **not** coalesce separate `VixDiskLib_Write` calls
(eight 8 KiB writes stayed eight IOs). One public write becomes N
client type-7 messages with the **same** `opId`, then **one** 44-byte
Expand All @@ -105,9 +106,9 @@ for one reply per chunk; raising the window did not match VDDK
throughput because VDDK pays one RTT per `Write`, not per fragment.

`NfcAioFlushCoalescedWrites` is server-side (`nfcAioServer.c`), not a
client merge of API writes. OPEN_SESSION is 16 zero bytes both ways, so
the logged AIO buffer count of 4 is a VDDK client default
(`vixDiskLib.nfcAio.Session.BufCount`), not a server cap.
client merge of API writes. Buffer count is
`vixDiskLib.nfcAio.Session.BufCount` (OPEN_SESSION offset 8); size is
`BufSizeIn64KB` (offset 4, in bytes). `docs/nfc_open.md`.

## OpenVixDiskLib

Expand Down
Loading
Loading