Skip to content

VMware to KVM: cold (VDDK) and warm (CBT) migration into Ceph/RBD and Linstor, plus importVm adoption of existing RBD root volumes and Linstor root/data volumes - #13656

Open
andrijapanicsb wants to merge 47 commits into
apache:mainfrom
andrijapanicsb:kvm-rbd-vmware-migration
Open

Conversation

@andrijapanicsb

@andrijapanicsb andrijapanicsb commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Description

This PR makes Ceph/RBD and Linstor first-class targets for KVM VM onboarding and adds an operator-driven warm VMware→KVM migration path. It bundles the following (Ceph/RBD plus Linstor as block-storage destinations):

1. VMware CBT → native RBD/qcow2 warm migration to KVM

An operator-driven, warm VMware-to-KVM migration built on VDDK and VMware Changed Block Tracking (CBT). It maintains a source-equivalent KVM-side replica kept current through repeated delta/sync cycles, then finalizes the guest with virt-v2v in-place at cutover — bounding downtime to the cutover window instead of a full cold copy.

  • Targets both filesystem (qcow2) and native Ceph/RBD (raw image) primary storage.
  • Full lifecycle as async operations with persisted migration state (migration / cycle / disk records), preflight validation, cancel/cleanup, and Import/Export UI integration.
  • New admin API commands: startVmwareCbtMigration, syncVmwareCbtMigration, cutoverVmwareCbtMigration, cancelVmwareCbtMigration, deleteVmwareCbtMigration, listVmwareCbtMigrations, checkVmwareCbtMigrationPrerequisites.
  • Server orchestration in VmwareCbtMigrationManagerImpl (+ VmwareCbtMigrationServiceImpl, cutover policy), backed by new VOs/DAOs and a DB schema migration.
  • KVM agent: libvirt CBT command wrappers (prepare / sync / cutover / RBD-probe) driving VDDK + qemu + virt-v2v, plus agent commands/answers and a sync-plan.
  • UI: new VMware CBT Migrations view and additions to Import/Manage Instances under Tools.
  • Docs: docs/vmware-cbt/ (architecture + README) and docs/vmware-cbt-migration.md.

2. Direct VDDK VMware import into Ceph/RBD — the cold (one-shot) path

The cold counterpart to the warm CBT migration in (1): extends the existing
VDDK + virt-v2v VMware→KVM import — which previously landed only on NFS/qcow2
(direct, or via a staged file) — so the converted disk is written straight into an
RBD pool as a raw image
, with in-place virt-v2v finalization and no intermediate
qcow2-on-NFS staging copy. (The warm path in (1) reuses this same full-disk copy for its
initial sync.) Adds host-capability detection for qemu-img RBD support, RBD qemu-copy,
virt-v2v in-place, and direct-RBD import — advertised via ReadyCommand and reconciled
by the agent manager.

3. Adoption of existing RBD (Ceph) and Linstor ROOT and DATA volumes

Extends the KVM unmanaged-import / manage-volume flow so existing RBD and Linstor
volumes can be adopted in place, without copying their data.

A ROOT volume is adopted with importVm importsource=shared. Additional DATA
volumes are discovered with listVolumesForImport, adopted with importVolume,
and attached with attachVolume.

For RBD, this adds RBD-aware format handling in VolumeOrchestrator (RBD ⇒ RAW
instead of the hardcoded QCOW2) and the RBD branch in the libvirt volume-inspection
path (LibvirtCheckVolumeCommandWrapper), so CheckVolumeCommand succeeds instead
of failing with "Disk not found or is invalid".

For Linstor (DRBD) primary storage, externally prepared ROOT and DATA volumes
(for example, landed by a migration/conversion tool) can be adopted without any
data copy. Linstor volumes are raw local block devices, so volume inspection runs
qemu-img against the /dev/drbd/… device path (LibvirtCheckVolumeCommandWrapper
for ROOT, LibvirtGetVolumesOnStorageCommandWrapper plus the volume
import/unmanage gate for DATA), and VolumeOrchestrator records RAW.

Because a host-local qemu-img file lock cannot see a DRBD resource that is
Primary/open on another node (unlike RBD's cluster-wide exclusive lock),
adoption additionally consults Linstor's cluster-wide InUse state — the new
KVMStoragePool.getVolumeInUseNode() capability (default no-op), answered via
the existing LinstorUtil.isResourceInUse — and refuses a volume attached to a
running VM anywhere in the cluster. Volumes are adopted under their existing
resource names (Linstor cannot rename; 48-character limit).

The existing filesystem/qcow2 adoption path is left unchanged, by
construction rather than by accident: the supported-pool-type list is extended
additively; the raw-device inspection branch is guarded by an explicit
RBD/Linstor check that returns early, so filesystem pools still fall through to
the original checkQcow2File path; the new cluster-wide in-use lookup is a
KVMStoragePool default returning null and is overridden only by
LinstorStoragePool; and the format resolution is a new overload that delegates
to the pre-existing method for every pool type other than RBD/Linstor (so
qcow2 pools still resolve to QCOW2).

Live coverage includes RBD and Linstor ROOT adoption through importVm, plus
RBD and Linstor DATA-volume adoption through importVolume. The adopted
volumes retained their original backend paths, were attached to booted VMs,
and preserved their checksums. The existing qcow2/NFS ROOT adoption path was
also rechecked live: the file was adopted in place, the VM booted, and the
pre-adoption checksum matched inside the guest.

4. Linstor (DRBD) primary storage as a migration destination

Extends all three VMware→KVM paths above — staged import, direct VDDK import, and warm CBT migration — so the converted disks can also land on Linstor primary storage, not just Ceph/RBD. A generic RAW_BLOCK_DEVICE target type covers "write RAW into a host-local block device provided by the storage adaptor", so the same code paths serve Linstor (and lay groundwork for other block backends).

  • The KVM agent pre-creates each Linstor resource at source capacity (unlike RBD, qemu-img cannot create a DRBD device), copies with qemu-img convert -n — or nbdcopy when the host has it — into the local /dev/drbd/… device, and finalizes with virt-v2v-in-place fed a <disk type='block'> domain XML (no qemu-nbd bridge needed for a local device).
  • Warm CBT delta cycles stream each changed extent from the nbdkit/VDDK source window straight into the same device window (qemu-img convert -n -S 0), avoiding a temp-file round-trip.
  • Server orchestration adds Linstor to the staged / direct-VDDK / CBT destination pool types and validates that the conversion host is a LINSTOR satellite connected to the pool.
  • LINSTOR resource-name limits (48 characters, no rename) are respected with short deterministic names; imported volumes keep the name as their recorded path.
  • Omits an unsupported -O flag from the virt-v2v-in-place invocation (fixes in-place finalization on virt-v2v < 2.5, for example Ubuntu 24.04) — which benefits the RBD path too.

5. Data-plane efficiency and correctness for the agent-driven copies

These apply to the agent's own nbdkit/VDDK full-disk copies (cold direct import and CBT initial full sync); they do not touch the staged virt-v2v conversion path.

  • VDDK NBD compression — the nbdkit VDDK plugin runs with compression=<value> (new vddk.nbd.compression agent property, default fastlz; none/zlib/fastlz/skipz). On datastores that do not expose block allocation to VDDK, the full nominal disk size otherwise crosses the network; fastlz compresses both zero regions and real data on the ESXi side at low CPU cost (measured on a 30 GiB disk: 32.4 GB → 2.65 GB over the wire).
  • Multi-connection nbdcopy for RBD full copies — cold-direct-to-RBD and the CBT initial full sync to RBD copy with nbdcopy over a localhost qemu-nbd bridge (pre-create the raw image, serve it locally, copy NBD-to-NBD) rather than a single-stream qemu-img convert, falling back to qemu-img convert when nbdcopy is unavailable. The Linstor block-device path uses nbdcopy directly against the local device; CBT delta cycles use a range copier.
  • Thin-safe zero handling — copies into a pre-created block-device target skip writing zero blocks (--target-is-zero / --destination-is-zero) only when the backend guarantees a freshly created volume reads back as zeros, via a new KVMStoragePool.isVolumeZeroInitialized() capability (Linstor answers it from the provider kind — thin providers only). On backends that do not guarantee it (for example LVM-thick) the zeros are written, so no stale data from a previously deleted volume leaks into the unwritten regions. qemu-img's --target-is-zero support is also probed directly (newer qemu dropped it from --help while still supporting it).

6. Windows guest post-migration handling

For Windows guests, a first-boot script is injected via virt-v2v --firstboot: on first boot it sets the SAN policy to OnlineAll and brings migrated data disks online and read-write, closing the common post-conversion gap where secondary disks come up Offline/Read-only under the default Windows SAN policy. It is best-effort and fail-safe — if it cannot run, the conversion still succeeds and the disks simply keep their default state. (Uses Storage cmdlets available on Windows Server 2012+.)

Networking. After conversion the guest keeps its source MAC (with forced=true semantics per the import flow), so on a CloudStack-managed network the virtual router hands the guest exactly the IP CloudStack allocated for that NIC. How that IP is chosen is covered by the static-IP preservation in (8) below.

7. Guest identity, firmware and console defaults derived from the source

Guest OS type and firmware (BIOS vs UEFI) are derived from the source VM's vCenter
configuration (config.guestFullName / config.guestId / config.firmware /
config.bootOptions) instead of relying on VMware Tools guest info. Tools is not running on a
powered-off source — and block-storage destinations require the source to be powered off — so
previously every such import silently landed on the import dummy template's defaults: a generic
Linux OS type and BIOS firmware. For a UEFI source that produces a VM that cannot boot at all
(GPT/EFI disk under SeaBIOS), and for Windows guests it additionally means wrong clock semantics
(UTC instead of localtime), no hyperv enlightenments, and — on Server Core — a console that renders
nothing on the legacy cirrus video device. Imports set the guest OS type (nearest catalog match
for releases newer than the catalog, for example Windows Server 2025 → 2022), the UEFI detail
(SECURE when the source has secure boot enabled), the q35 machine type for UEFI sources, and a
vga console for Windows guests. Explicit caller-supplied values always take precedence. The
conversion fails fast with a clear message when virt-v2v reports that the host's
virtio-win driver set has no drivers for the guest's Windows release, instead of delivering an
unbootable VM. Related platform issue for the cirrus default affecting native deployments too:
#13806.

8. Static source IPv4 preservation

When the operator does not supply an explicit nicipaddresslist entry for a NIC, the import
keeps the guest's own IPv4 address rather than allocating a random one. While the source
runs, per-NIC addresses with prefixes, the subnet's default gateway and the DNS servers are captured
from VMware Tools (GuestNicInfo.ipConfig, with a fallback to the bare address list for older
Tools). At import time each unspecified NIC keeps its captured address if it fits the target
network's CIDR, is not the gateway and is not already in use; otherwise — and for NICs with no
usable source data, for example DHCP guests whose lease belongs to the source network — the NIC falls back
to automatic allocation (the "auto" marker, resolved by the network orchestrator to a free
address). An explicitly supplied nicipaddresslist entry always wins. The result: a static Windows
guest arrives on KVM with the same address inside the guest and in CloudStack's allocation,
with no operator input and no in-guest rewriting.

9. Hardening from extended lab testing

  • Start-time validation and terminal-state handling for the CBT migration lifecycle are strict:
    consistent rejections, and no records stranded in a non-terminal state after a failure.
  • The agent's qemu-nbd bridge port selection holds all probed ports until every bridge port is
    chosen, so concurrent disk bridges cannot race for the same port.
  • The Linstor destination fails closed when the cluster state cannot be established, rather than
    proceeding on incomplete information.

Types of changes

  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Bug fix
  • Enhancement

Feature/Enhancement Scale

  • Major
  • Minor

How Has This Been Tested?

Unit tests

Unit tests cover all new and touched wrappers, managers and orchestration,
including LibvirtImportConvertedInstanceCommandWrapperTest,
LibvirtConvertInstanceCommandWrapperTest,
LibvirtCheckConvertInstanceCommandWrapperTest,
LibvirtCheckVolumeCommandWrapperTest, the LibvirtVmwareCbt* suites,
UnmanagedVMsManagerImplTest, VolumeImportUnmanageManagerImplTest, the
server-side VmwareCbt* suites and VolumeOrchestratorTest.

Final end-to-end acceptance run

The final end-to-end acceptance run used management and KVM packages built from the exact commit published as the current PR head:

f8249625cff44ea15de7652f50a568c967c6c23c

No uncommitted changes or additional unpublished code were included in the tested package set.

All API/CLI testing used authenticated CloudMonkey requests on port 8080. The
unauthenticated integration API was not used.

Scope Completed Passed Blocked Failed
Automated API/CLI cases 32 / 32 32 0 0
Supervised canonical UI cases 4 / 4 3 1 0
Canonical acceptance total 36 / 36 35 1 0
Final cleanup and health gate 1 / 1 1 0 0
Supplemental warm-CBT UI documentation 1 / 1 1 0 0

The one blocked case is the project-network selection in the import wizard.
The project's network was not listed, so no import was submitted and no test
resource was created. This is a separately tracked upstream UI problem, not a
PR #13656 functional failure. Authenticated project ownership and isolation
passed through the API/CLI case O03.

The final cleanup/health gate and the supplemental UI documentation run are
outside the canonical 36-case denominator.

Functional coverage

Area Cases Result Coverage
Storage topology P01-NFS-SMOKE, P01-RBD-SMOKE, P01-LINSTOR-SMOKE PASS Pool attachment, backend health and deterministic placement
Native controls N00, C00, L00 PASS Native Ubuntu lifecycle and storage I/O on NFS, RBD and Linstor
Cold staged import N01, S01, S02 PASS Staged import on all three backends and temporary-file cleanup
Direct VDDK import N02, C01, L01 PASS Windows on NFS and dual-disk Ubuntu on RBD and Linstor
Warm CBT migration N03, C02, L02 PASS Manual cutover on NFS, RBD and Linstor
Cancel during replication N04 PASS Source snapshot and destination cleanup
Storage migration C03 PASS NFS → Ceph → NFS with volume mapping and checksums preserved
Negative validation B01, B02, V01, V02 PASS Powered-on import, undersized offering and incomplete NIC mapping rejected cleanly
Resilience and retry R01R05 PASS Cancel, invalid state transitions, final-import retry and failed-record cleanup
Existing-volume adoption A01, A02, A03 PASS NFS qcow2, RBD and Linstor DATA volumes adopted in place, attached, booted and checksum-verified
Authenticated ownership O01, O02, O03 PASS Admin, account/domain and project ownership with isolation

ROOT adoption through importVm importsource=shared was already live-validated
for RBD and Linstor, including adopt-and-boot. The final regression run
revalidated the companion DATA-volume path through importVolume on RBD and
Linstor (A02 and A03). Direct and warm migration of ROOT + DATA disk sets was
additionally covered by C01, L01 and L02; those cases exercise migration,
not adoption.

Guest boot was accepted only after guest-level evidence. Where applicable, the
run also verified disk ordering, storage placement, source and destination
checksums, network reachability, stop/start/reboot lifecycle, source-snapshot
cleanup and the absence of temporary conversion data.

Supervised UI coverage

Case Result Evidence
UI01 — admin import PASS Correct mapping, admin ownership, two disks, network, guest validation and cleanup
UI02 — account/domain import PASS Correct VM, volume and network ownership plus unrelated-account isolation
UI03 — project import BLOCKED Project network not listed; no submission and no resource created; O03 passed independently
UI04 — unauthorized-network negative PASS Server returned permission denied; no VM, volume or migration artifact was created

A supplemental UI run (UI05) captured the complete warm-CBT workflow:
InitialSync, explicit changed-block delta cycles, ReadyForCutover, manual
source shutdown, final cutover, in-place conversion and Completed. It recorded
three delta cycles and 59,834,368 changed bytes in total. The completed VM and
both volumes were verified on KVMPrimary and then fully cleaned up.

Warm CBT UI evidence

Ready for the operator-approved final cutover after two completed delta cycles:

VMware CBT migration ready for cutover

Completed migration after the final delta synchronization and in-place conversion:

VMware CBT migration completed

Final environment state

  • Both routing hosts were Up and Enabled.
  • All four primary pools were Up; the image store was present.
  • SSVM and CPVM were Running with agents Up.
  • Ceph reported HEALTH_OK, 3/3 OSDs up/in and all placement groups clean.
  • Linstor reported HEALTHY with the expected nodes and pools online.
  • No suite-owned VM, active volume, snapshot, migration, worker, NFS path, RBD
    image, Linstor resource or source clone remained.

Earlier targeted validation retained

In addition to the final acceptance run, this PR was previously validated with:

  • a vSphere 8.0.3 source as well as the vSphere 7 environment used for the final run;
  • Windows Server 2012 R2 and Windows Server 2025 cold and warm migration;
  • source-derived UEFI/q35/video/guest-OS defaults and supported KVM CPU models;
  • static IPv4 preservation for static, DHCP and foreign-subnet NIC combinations;
  • RBD and Linstor ROOT adoption, Linstor ROOT + DATA adopt-and-boot, and qcow2/NFS ROOT adoption;
  • the nbdcopy full-copy path and the qemu-img fallback path.

Full acceptance report

PR13656-acceptance-report.pdf

Operator-driven warm VMware-to-KVM migration built on VDDK and VMware
Changed Block Tracking (CBT). Keeps a source-equivalent KVM-side replica
current through delta cycles, then finalizes with virt-v2v in-place at
cutover. Supports both filesystem (qcow2) and native Ceph/RBD (raw image)
primary storage targets, with preflight validation, async lifecycle
operations, persisted migration state, and Import/Export UI integration.

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Extends the existing VDDK + virt-v2v VMware-to-KVM import path so the
converted disk can be written straight into an RBD pool as a raw image,
with in-place virt-v2v finalization. Adds host-capability detection for
qemu-img RBD support, RBD qemu copy, virt-v2v in-place, and direct RBD
import, advertised via ReadyCommand and reconciled by the agent manager.

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Extends the KVM unmanaged import / manage-volume flow so an existing raw
RBD image can be adopted directly as a VM root volume, including the
RBD-format check in the volume orchestrator and the libvirt volume
inspection path.

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Unifies the host-capability keys that the CBT warm-migration and VDDK
direct-RBD-import features introduced independently under different
names, using feature-neutral names that describe what is probed:
  host.vmware.cbt.support                     -> host.vddk.blockcopy.support
  host.vmware.cbt.in.place.finalization.support -> host.vddk.blockcopy.inplace.finalization.support
  host.vmware.cbt.rbd.support                 -> host.vddk.blockcopy.rbd.support
  host.qemu.img.rbd.support                   -> host.qemu.rbd.support
  host.virt.v2v.inplace.support / host.virtv2v.in.place.version -> host.virtv2v.inplace.*
Drops the redundant host.rbd.qemu.copy.support alias (identical to
host.qemu.rbd.support) and its hostSupportsRbdQemuCopy() delegate.

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
@andrijapanicsb andrijapanicsb added this to the 4.24.0 milestone Jul 21, 2026
@andrijapanicsb andrijapanicsb changed the title KVM/VMware: warm CBT migration and direct VDDK import into Ceph/RBD, plus importVm adoption of existing RBD root volumes KVM/VMware: warm/change tracking migration and direct VDDK import into Ceph/RBD, plus importVm adoption of existing RBD root volumes Jul 21, 2026
andrijapanicsb and others added 8 commits July 21, 2026 06:58
…isk import

The "Manage Instances" source dropdown offered "Import QCOW2 image from
Local/Shared Storage", but importVm now adopts an existing ROOT disk from
Filesystem, NetworkFilesystem, SharedMountPoint and RBD (raw) pools, so the
QCOW2-only wording is misleading. Rename both entries and their wizard
title/description to "Import Instance using existing ROOT disk from
Local/Shared Storage", across the en, pt_BR and te locales.

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
docs/vmware-cbt/architecture.md and docs/vmware-cbt/README.md were added
without the Apache license header, which failed the RAT license check and the
pre-commit insert-license hook. Add the standard ASF header (identical to the
one already present in docs/vmware-cbt-migration.md).

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
…ey names

State Apache CloudStack 4.24.0.0 as the target release (schema delivered via the
4.24.0.0 upgrade path, with a note that the DDL currently rides in
schema-42210to42300.sql pending the main version bump), and replace the
pre-consolidation host-capability key names with the current
host.vddk.blockcopy.* / host.virtv2v.inplace.version names.

Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Staged VMware-to-KVM imports (virt-v2v conversion to an NFS temporary
location followed by a copy to the destination pool) can now target
Linstor primary storage:

- Linstor added to the staged-conversion destination pool types, so
  destination pool discovery considers cluster- and zone-wide Linstor
  pools alongside NFS and RBD.
- The import host is validated to have access to each Linstor
  destination pool (it must be a LINSTOR satellite connected to the
  pool); the RBD qemu-copy capability check is preserved and the
  validation renamed to validateStagedImportHostSupport.
- The KVM-side converted-disk move already dispatches by pool type to
  LinstorStorageAdaptor.copyPhysicalDisk, which spawns the resource
  from the pool's resource group and qemu-img converts the qcow2 into
  the DRBD block device; converted-disk metadata for Linstor pools now
  reports the pool UUID and volume name instead of parsing NFS mounts
  (Linstor pools have no local mount path).

The imported volume keeps the copy-time UUID as its path; the Linstor
driver addresses existing volumes by path, so later volume lifecycle
operations (attach, resize, snapshot, delete) work unchanged.
Extends the direct (non-staged) VDDK import path, previously available
only for Ceph/RBD, to Linstor primary storage. With usevddk=true and
forceconverttopool=true on a Linstor conversion pool, the KVM host
copies each VMware disk over nbdkit/VDDK straight into a pre-created
DRBD block device and finalizes it with virt-v2v in place - no NFS
staging and no intermediate qcow2 copy.

Server side:
- Linstor added to the VDDK direct-conversion pool allow-list; the
  RBD/Linstor "requires usevddk" validation is shared.
- Conversion host selection for Linstor requires VDDK support,
  in-place virt-v2v support, and access to the Linstor pool (the host
  must be a LINSTOR satellite); explicit host selections are validated
  for pool access as well.
- CheckConvertInstanceCommand carries a new in-place finalization
  check so unsupported hosts fail fast with a clear message.
- The powered-off, non-cloned source VM requirement now applies to any
  direct block-storage import.

KVM agent side:
- The direct conversion routine handles both targets: for Linstor it
  pre-creates each resource at the source capacity through the storage
  adaptor (qemu-img cannot create DRBD devices, unlike RBD images),
  copies with qemu-img convert -n to the local DRBD device, and feeds
  virt-v2v-in-place a <disk type='block'> domain XML - no qemu URI or
  RBD-style network disks needed.
- Storage access is probed with a 4 MiB test volume and a qemu-io
  write/read round-trip before any data transfer.
- Linstor disk names use a shorter '-dNN' suffix: LINSTOR resource
  names are capped at 48 characters, which the RBD '-disk-NNN' scheme
  would exceed.
- The forced-conversion import short-circuit accepts Linstor temporary
  locations, backed by a new LinstorStorageAdaptor.listPhysicalDisks
  implementation (resource definitions filtered by the pool's resource
  group), replacing the previous UnsupportedOperationException.
Extends the VMware CBT warm-migration framework, previously limited to
qcow2 file and Ceph/RBD targets, with a generic raw-block-device target
type backed by Linstor primary storage. The initial full sync, the
incremental CBT delta cycles and the powered-off cutover all write
straight into the DRBD block device of a pre-created Linstor volume.

Target classification and orchestration:
- New VmwareCbtTargetStorageType.RAW_BLOCK_DEVICE; Linstor pools
  classify as supported block-device targets that require in-place
  virt-v2v finalization (the qcow2 fallback cannot write to devices).
- Linstor added to the CBT-compatible pool types for explicit and
  implicit destination pool selection.
- Conversion host selection requires VMware CBT migration support,
  in-place finalization support and access to the Linstor pool (the
  host must be a LINSTOR satellite); both auto-selection filtering and
  explicit host validation enforce it.
- The destination storage probe now covers block-device pools: the
  agent creates a 4 MiB volume through the storage adaptor, round-trips
  a qemu-io write/read on the local device and deletes it.

Naming: block-device target volumes are named cbt-<mig8>-<diskId>,
where mig8 is the first eight characters of the migration UUID. LINSTOR
resource names are limited to 48 characters (the volume name gets a
cs- prefix as a LINSTOR resource), so the RBD-style names carrying the
full migration UUID do not fit. The short marker keeps cleanup guarded:
both the management server and the agent only delete volumes whose
names start with the marker derived from the migration UUID.

Agent-side data path:
- Initial sync pre-creates each Linstor volume at source capacity via
  the storage adaptor (qemu-img cannot create DRBD devices) and runs
  the nbdkit/VDDK captive copy with qemu-img convert -n to the local
  device path.
- Delta cycles resolve the device path and patch changed extents with
  the existing qemu-io write mechanism; raw device targets are accepted
  by disk validation.
- Cutover feeds virt-v2v-in-place a <disk type='block'> domain XML with
  the local device path - the qemu-nbd localhost bridges required for
  RBD are unnecessary for block devices.
- Cleanup deletes only marker-guarded volumes through the adaptor.

The finalized volumes are imported through the existing shared path;
imported Linstor volumes keep the target name as their volume path,
which the Linstor driver uses to address existing volumes.
Covers the release note for the three import modes (staged, direct
VDDK, CBT warm migration), the operator guide's destination-specific
host requirements and operational notes (satellite/node naming, single
Primary writer, diskful replica preference, DRBD size rounding,
heartbeat fencing interaction, retained volume names), and the
as-built design guide's storage-target section (RAW_BLOCK_DEVICE
mapping, pre-created devices, convert -n, block-disk XML finalization,
48-character resource-name constraint and the short cleanup marker).
Found during live validation on an Ubuntu 24.04 LINSTOR lab node:
virt-v2v-in-place 2.4.0 does not know the -O (write updated output
XML) option, which only exists from virt-v2v 2.5 on (EL9 ships 2.5.x,
where the RBD flows were originally validated). Both the VMware CBT
cutover finalization and the direct VDDK import finalization passed -O
unconditionally, so otherwise fully capable hosts failed the in-place
step at the very end of a migration.

Nothing ever consumed the output XML - both wrappers created a
temporary file for it and deleted it unread - so the option and the
temporary files are removed rather than gated on a version probe.
Verified live: virt-v2v-in-place 2.4.0 with the same libvirt XML
converts a guest on a DRBD block device successfully without -O.
@andrijapanicsb
andrijapanicsb requested a review from abh1sar August 3, 2026 15:45

@rp- rp- left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its fine from my side, there is 1 thing I'll harden in the deRefOrDeleteResource method in Linstor. As it might now be possible that resource/cs paths have now different length. But I'll make sure this lands before the next CS release

Zero-block detection and the cluster-wide in-use check both treated an
unreachable or empty controller response the same as a healthy answer.
Zero-copy optimizations could then skip writing zeros into recycled
extents, and volume adoption could treat a volume attached elsewhere as
free. Both paths now fail closed: unknown zero-state disables the
zero-skip, and an unknown in-use state is reported through the new
KVMStoragePool.IN_USE_NODE_UNKNOWN marker which the import wrappers
treat as locked.
… is chosen

The RBD cutover pre-allocated one localhost port per disk by opening and
immediately closing a probe socket in a loop, so the kernel could hand
the same ephemeral port to two disks of the same migration. The probe
sockets are now kept open until all ports are selected and released only
right before the bridge script starts qemu-nbd.
Three related server-side gaps:

- A cancelled migration could be resurrected by an in-flight replication
  or cutover job writing its progress back over the terminal state,
  leaving a record that referenced already-deleted target disks and
  could never be deleted. State updates from long-running jobs now go
  through a single conditional DAO update that refuses to overwrite a
  terminal state, and cancellation marks the record cancelled before it
  removes the target disks.

- Data-disk offering mappings were only counted at start. They are now
  validated by identity: unknown disk ids and extra mappings are
  rejected, and each mapped disk offering goes through an access check
  for the owning account.

- NIC mappings were not validated at start at all, so a warm migration
  with missing or unknown NIC mappings only failed at import time, after
  full replication. The source NICs are captured during preflight and
  the requested mappings are validated before the migration record is
  created.
…VM configuration

An imported VM whose guest OS or firmware was not chosen by the caller
inherited the import dummy template's generic OS type and the KVM
defaults: BIOS firmware and libvirt's legacy cirrus video. A UEFI
source then cannot boot at all (no loader for a GPT/EFI disk under
SeaBIOS), and a Windows guest gets Linux clock semantics, no hyperv
enlightenments, and - on Server Core - a console that renders nothing.

The source VM's identity and firmware are available from the vCenter
configuration regardless of power state (config.guestFullName,
config.guestId, config.firmware, config.bootOptions) and were already
carried on UnmanagedInstanceTO; the import paths simply never used
them. Both the cold importVm path and the warm CBT migration now:

- resolve the CloudStack guest OS type from the source identity: exact
  display name with and without the vendor prefix, then the VMware
  guest id mapping, then for Windows releases newer than the catalog
  the nearest same-family entry (the display name must keep the
  Windows prefix, which the KVM agent keys clock and hyperv behaviour
  on), and
- fill the hardware details from the source: UEFI (SECURE when the
  source has secure boot enabled) plus the q35 machine type for UEFI
  sources, and a vga console for Windows guests.

Caller-provided values always take precedence; only absent parameters
are derived.

Also reject datadiskofferinglist on shared/local storage imports
instead of silently ignoring it: that path only adopts the root disk,
and accepting the parameter let callers believe their data disks were
imported. Data disks are adopted with importVolume and attached.
…available

virt-v2v treats a missing virtio driver set as a non-fatal warning and
completes the conversion, but a Windows guest converted without a
virtio storage driver cannot boot from the virtio disk the import
attaches - the failure only surfaced later as an unbootable VM with no
hint of the cause (typically an enterprise-distro virtio-win package
that predates the guest's Windows release). The conversion output is
now watched for virt-v2v's warning and a Windows conversion fails
immediately with a message that names the host's virtio-win driver set
as the thing to update.
Operators had to read each source VM's addresses by hand and pass them
through nicipaddresslist, or the imported VM came up with a
CloudStack-allocated IP that did not match the guest's own static
configuration.

The guest network facts are now captured per NIC from VMware Tools
while the source runs - IPv4 addresses with prefix length from
GuestNicInfo.ipConfig (the plain address list has no prefix), plus the
default gateway and DNS servers from the per-stack route and DNS
configuration, with each gateway matched to the NIC whose subnet
contains it. The facts travel on UnmanagedInstanceTO.Nic and on the
warm-migration preflight NIC info.

At import (cold) and at warm-migration start, a NIC that is mapped to
a network but has no caller-provided IP keeps the guest's own address
when it fits the target network's CIDR, is not the network's gateway
and is not already in use; otherwise it falls back to normal
allocation. Caller-provided entries always take precedence.
…nfig is absent

Older and limited VMware Tools versions report guest addresses only as
a plain list without ipConfig, so a capture that reads only ipConfig
silently produces nothing for those guests. IP preservation needs just
the address itself - the fit test runs against the target network's
CIDR - so the bare list is used when ipConfig gave nothing.
…irtio-win fail-fast

Document the behavior added in this change set: guest OS type, firmware,
machine type and Windows console video derived from the source VM's
configuration; static IPv4 preservation from VMware Tools with its
fit/fallback rules; and the fail-fast when virt-v2v has no virtio
drivers for a Windows guest, including the virtio-win ISO guidance and
the Windows Server 2025 smbus caveat.
…annot be preserved

When no explicit NIC IP is given and the source address cannot be preserved
(missing Tools data, address outside the target network CIDR, gateway clash or
address already in use), the import into an isolated or shared network used to
fail with "NIC needs a valid IP address". Fill in the "auto" marker instead, so
the network orchestrator picks a free address for that NIC. Lab-validated with
a two-NIC Windows guest (static NIC preserved, DHCP NIC auto-allocated).
@apache apache deleted a comment from ACSHomeBot Aug 7, 2026
@apache apache deleted a comment from ACSHomeBot Aug 7, 2026
@apache apache deleted a comment from ACSHomeBot Aug 7, 2026
@apache apache deleted a comment from ACSHomeBot Aug 7, 2026
@ACSHomeBot

Copy link
Copy Markdown

PR package build

Commit: 73c6847f4422660eda32488636f47a061126e6ca

Result Artifact Platforms / target
PASS RPM EL9 build target
PASS DEB Ubuntu, Debian
PASS SystemVM template KVM only

Internal artifacts are retained as current and previous for this PR; the private base URL is intentionally not published.

@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@ACSHomeBot package kvm

@ACSHomeBot

ACSHomeBot commented Aug 7, 2026

Copy link
Copy Markdown

Packaging results (commit 73c6847)

Result Artifact Platform
PASS RPM EL (EL8/9/10)
PASS DEB Ubuntu, Debian

Test packages are available at:

Those releases need CPU features the KVM agent's default guest model (qemu64)
does not expose. The conversion works, the instance starts and applies its UEFI
firmware, but the guest then enters "Preparing Automatic Repair" instead of
reaching the login screen, and nothing in the migration output hints at why.

Document setting guest.cpu.mode=custom with a named model, and tell the reader
to pick one the host actually supports by checking virsh domcapabilities first.
The right model differs by processor: Broadwell-noTSX-IBRS or newer on Intel,
EPYC-IBPB or newer on AMD. Setting a model the host reports as unusable is worse
than leaving the default, because every newly started domain then fails with
"CPU is incompatible with host CPU", system VMs included, while already running
domains carry on as if nothing is wrong.

This is a host prerequisite for the guest OS, not a property of the migration.
It applies the same way to instances deployed natively on KVM.
…is cancelled or fails

Successful migrations remove the baseline snapshot they created on the source VM,
but a cancelled or failed one left it behind. The removal was attempted, but right
after a cancellation the source is often still serving the transfer that was just
torn down, vCenter answers "The operation is not allowed in the current state", and
the single attempt gave up for good. The snapshot then stays on the customer's
production VM, silently growing a delta disk, with nothing in CloudStack showing it
exists. Observed live twice: one snapshot survived a successful retry, a second
cancel and the record deletion; another from a failed conversion was still there a
day later.

Retry the removal a few times while vCenter reports the source busy, and sweep any
snapshot still recorded against the migration's disks when the migration is
cancelled and when it is deleted with cleanup. The cancel-path sweep runs only
after the target cleanup command has returned, because that is what stops the
nbdkit readers holding the disk open; the delete-path sweep runs before the source
credentials are cleared and the disk rows removed, since after either the snapshot
can no longer be found or authenticated for. If every attempt fails, say plainly in
the log that the snapshot is still on the source VM and must be removed manually.
@JoaoJandre
JoaoJandre removed their request for review August 10, 2026 19:41
@github-actions

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

Functional acceptance is complete. The PR has also been approved by two independent Apache CloudStack committers.

This PR targets CloudStack 4.24 and must not be merged into 4.23. Please hold the merge until main is bumped to 4.24.0.0-SNAPSHOT. After the version bump, the database changes will be moved to the official 4.23.0 to 4.24.0 upgrade path, the API since values will be aligned with 4.24, and the branch will be refreshed to resolve the current merge conflict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants