RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype

[zidz]

TL;DR

I built an incremental RBD backup provider for CloudStack (port to 4.23.0.0).
It uses Ceph RBD snapshots + export-diff/import-diff so an incremental backup
is ~32x smaller than the full, and a restore replays the full→incremental chain.
Full + incremental + restore are verified end-to-end through the management API
against a real Ceph/KVM host. This is a prototype for design feedback, not a
finished PR — I'd like input on two open questions below.

Motivation

The existing NAS backup provider copies whole disk images every time. On Ceph
RBD primary storage we can do far better: take an RBD snapshot per backup and
export only the diff since the previous snapshot. Backups become small and fast,
and a restore is import (base) + import-diff (chain).

Design

• New plugin plugins/backup/rbd implementing BackupProvider.
• Full = rbd snap create + protect + export → <disk>.<vol>.rbd.
• Incremental = export-diff --from-snap <parent> → <disk>.<vol>.<ts>.rbdiff.
• Restore = import base + import-diff each link, truncated at the
  selected backup (restoring an earlier backup lands on that point-in-time).
• Chain metadata (chain id, position, parent snap) stored as backup details.
• Host-side engine: scripts/vm/hypervisor/kvm/rbdbackup.sh (called by the KVM
  agent wrappers, mirroring the NAS path).

What changed vs 4.23.0.0

• New (additive): plugins/backup/rbd/** (provider + tests + spring module),
  scripts/vm/hypervisor/kvm/rbdbackup.sh.
• Existing code (minimal, null-guarded):
  • core/.../backup/{Take,Delete,Restore}BackupCommand.java — added RBD fields
    (parent snap, base snap, rbd uri/snap, disk types). New Command fields only;
    rolling-upgrade safe (old agents ignore unknown fields).
  • kvm/.../Libvirt{Take,Delete,Restore}BackupCommandWrapper.java — route
    pure-RBD VMs to rbdbackup.sh; NAS path unchanged (guarded branch).
  • plugins/pom.xml + debian/rules — build & package the new module.

Enabling it today

createBackupOffering is currently hard-gated to the KBOSS provider, and
there's no createBackupRepository API, so the offering + repository must be
created via DB for now:

• set backup.framework.enabled=true, backup.framework.provider.plugin=rbd,
  rbd.backup.incremental.enabled=true
• insert a backup_repository (NFS) and a backup_offering (provider=rbd),
  linked by backup_offering.external_id = backup_repository.uuid
  Full step-by-step is in plugins/backup/rbd/README.md.

Verification (real Ceph/KVM host)

Scenario	Result
Full backup	.rbd object, ~5 GB
Incremental (no disk writes)	.rbdiff, ~132 B
Incremental (512 MB written)	.rbdiff, ~512 MB
Restore #1 / #2 / #3 of a 3-backup chain	lands on the correct point-in-time state each time
Restore-to-existing-VM	purges dest image, replays chain, data intact
restore-as-template	rebuilds chain into a new, freely-named image

Limitations / known issues

• Restore is bound to the original VM (framework limitation — restoreBackup
  resolves the VM from backup.getVmId()). For a new, custom-named VM use
  restore-as-template → register as template → deploy.
• Offering/repository creation is manual (KBOSS-gated API) — see open question 1.
• The new module is always built/loaded (plugins/pom.xml, debian/rules), so a
  defect there has global blast radius — see open question 2.

Open questions for the community

1. Offering API: should createBackupOffering be generalized beyond KBOSS so
   RBD (and other providers) can create offerings via API/UI, removing the manual
   DB step? Or is there a preferred existing mechanism I missed?
2. Packaging: is always-on module registration acceptable, or should the RBD
   provider be opt-in (profile/flag) to limit build/startup coupling?

Try it

Branch feature/rbd-incremental-backup-4.23 (6 commits on top of 4.23.0.0): compare link
Docs: plugins/backup/rbd/README.md.

Feel free to get inspiration from, use full or parts of this implementation if anyone have use for it in any way.

[weizhouapache]

@zidz
thanks for the sharing.
If I understand correctly, the same ceph RBD pool will be used as the backup repository, right ?

for your information, I am working on incremental backup of RBD volumes to NAS backup right now.
the code has some conflicts with #14194 and #14203, I will maybe wait until the two PRs to be merged.

[zidz]

The backup will be stored in the secondary storage.
If you are working on it, please go ahead. My fork is more of a working proof of concept to put the idea and examples out there.

[weizhouapache]

The backup will be stored in the secondary storage. If you are working on it, please go ahead. My fork is more of a working proof of concept to put the idea and examples out there.

@zidz
IMHO, all changes should be put in plugins/backup/kboss/. no extra jar is needed.
@JoaoJandre @winterhazel can give some advise

[StepBee]

@zidz @weizhouapache Thats awesome work from both of you, exactly what i've been looking for.

At the moment i'm solving the requirement for incremental backups based on Ceph snapshot-diffs using external scripts - which unfortunately have no integration with Cloudstack so far.

So i am very excited about the work of both of you, thanks a lot!