VMBackup HotplugFailed on VMs with Fedora/RHEL DataSource: "disk.img: no such file or directory"

[shubham-pampattiwar]

Problem

When performing a backup of a VM provisioned from a Fedora DataSource (using dataVolumeTemplates with sourceRef), the KubeVirt VMBackup hotplug step fails with:

Warning  HotplugFailed  virtualmachineinstance/test-vm  
  failed to mount filesystem hotplug volume vmb-...-backup-target-pvc: 
  lstat /proc/1/root/var/lib/kubelet/plugins/kubernetes.io/csi/
  openshift-storage.rbd.csi.ceph.com/.../disk.img: no such file or directory

The VMBackup still reports Done: True despite this failure, resulting in an empty backup target PVC (directory structure exists but no qcow2 files). The datamover pod then fails with no qcow2 files found in /backup-data.

Environment

• OpenShift CNV v4.99.0-0.1771785652
• Storage: ocs-storagecluster-ceph-rbd
• Feature gates enabled: IncrementalBackup, UtilityVolumes, HotplugVolumes

What works vs. what doesn't

VM	Root Disk	Machine Type	Firmware	Hotplug	Backup
cirros (containerdisk source, 150Mi, RWO Block)	Small containerdisk	q35	BIOS	Works	Works
test-vm (Fedora DataSource, 30Gi, RWX Block)	Large DataSource	pc-q35-rhel9.2.0	EFI + SMM	HotplugFailed	Empty PVC

Both VMs have changedBlockTracking: "true". Both backup target PVCs are created identically (10Gi, RWO, Filesystem mode, same storage class).

How the backup target PVC is created

Our datamover controller creates the backup target PVC via ensureTempPVC() — a plain Filesystem PVC with no special configuration:

pvc := &corev1.PersistentVolumeClaim{
    ObjectMeta: metav1.ObjectMeta{
        Name:      pvcName,      // "kubevirt-backup-<du.Name>"
        Namespace: namespace,    // VM namespace
    },
    Spec: corev1.PersistentVolumeClaimSpec{
        AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce},
        Resources: corev1.VolumeResourceRequirements{
            Requests: corev1.ResourceList{
                corev1.ResourceStorage: resource.MustParse("10Gi"),
            },
        },
        // No volumeMode specified (defaults to Filesystem)
        // No storageClassName specified (uses cluster default)
    },
}

Question for KubeVirt team: Should the backup target PVC be created differently? Does the VMBackup API expect a specific volumeMode, annotation, or pre-existing disk.img file?

Observed behavior

1. Controller creates temp PVC kubevirt-backup-<name> in VM namespace (Filesystem, 10Gi, empty)
2. VMBackup CR is created with forceFullBackup: true
3. virt-handler logs "successfully mounted" at mount.go:561 (Kubernetes-level PVC attachment succeeds)
4. HotplugFailed: virt-handler's filesystem hotplug into the VM fails — looks for disk.img inside the freshly provisioned empty PVC
5. VMBackup reports VirtualMachineBackupCompletedSuccessfully / Done: True despite the hotplug failure
6. QEMU never receives the backup volume — virt-launcher logs show zero backup activity (no backup commands, no NBD export, no volume attachment inside the VM)
7. PV is rebound to OADP namespace, datamover pod mounts it
8. Debug pod inspection confirmed the PVC is empty — the checkpoint directory structure exists (test-vm/<checkpoint-name>/) but contains no files at all
9. Datamover pod fails: no qcow2 files found in /backup-data

Reproduced consistently across 4 backup attempts (b, c, d, e) for test-vm.

Key observations from investigation

The disk.img expectation

KubeVirt's filesystem PV disk documentation states that for standard filesystem hotplug volumes, a disk.img file must exist inside the PVC (auto-created for regular VM disks). However, the backup target PVC is not a VM disk — it's a destination for qcow2 export data. Nobody creates disk.img in it, and the hotplug code doesn't distinguish between a regular disk PVC and a backup target PVC.

Two-step mount process

The mount at mount.go:561 (Kubernetes-level PVC attachment to the virt-launcher pod) succeeds for both VMs. The failure happens at the second step — virt-handler's filesystem hotplug into the VM — which tries to bind-mount disk.img from the PVC into QEMU. This second step succeeds for cirros but fails for test-vm, despite identical backup target PVC setup.

Mount duration difference

The backup volume mount duration differs between the two VMs:

• cirros: mounted for ~1 second before unmount
• test-vm: mounted for ~33 seconds before unmount

UtilityVolumes feature gate

The UtilityVolumes feature gate is enabled on the cluster. KubeVirt introduced UtilityVolumes (PR #15922) specifically for backup PVC attachment — UtilityVolumes mount as a directory (no disk.img needed) rather than using the standard filesystem hotplug path. However, the VMBackup controller appears to attach the backup PVC as a regular hotplug volume rather than via spec.utilityVolumes on the VMI.

VMBackup completion status is misleading

• cirros: VirtualMachineBackupCompletedWithWarning (guest agent not connected) — backup data was written
• test-vm: VirtualMachineBackupCompletedSuccessfully — but no backup data was written at all

The "successful" completion is incorrect. The VMBackup should detect the HotplugFailed condition and report failure.

Questions for KubeVirt/CNV team

1. Why does hotplug work for cirros but fail for test-vm? Both backup target PVCs are identical. The VM configuration differences (EFI firmware, machine type pc-q35-rhel9.2.0, disk size/access mode) may influence which hotplug code path is taken.

2. Should the backup target PVC use UtilityVolumes? The UtilityVolumes feature gate is enabled but the VMBackup controller doesn't seem to use it. Is this expected for this KubeVirt version?

3. Should we create the backup target PVC differently? Does it need Block volumeMode, a pre-created disk.img, or specific annotations?

4. VMBackup should fail when HotplugFailed occurs. Currently it reports Done: True with no data written. This makes it impossible for downstream consumers to detect the failure from the VMBackup status alone.

How to reproduce

1. Deploy a Fedora VM from the fedora DataSource in openshift-virtualization-os-images with changedBlockTracking: "true", EFI firmware, pc-q35-rhel9.2.0 machine type
2. Trigger a Velero backup with snapshotMoveData: true
3. Observe HotplugFailed events: oc get events -n <vm-namespace> --field-selector reason=HotplugFailed
4. Inspect the backup PVC contents — checkpoint directory will be empty
5. Observe datamover pod failure: no qcow2 files found in /backup-data

[weshayutin]

@ShellyKa13 @mhenriks FYI

[ShellyKa13]

Hi so we have several upstream tests that run with Fedora images and work just fine, so I will assume its something in the cluster or in the configurations that is not right.
The hotplug of the utilityvolume pvc should not expect to have disk.img. The pvc should be filesystem PVC not block.
Also the backup should not have proceeded unless the attachment succeeded and the volume status in the vmi status showed that the pvc is mounted.
I will appreciate either an env that I can look and investigate or also the vm vmi yamls with the status and conditions during the backup, also the backup yaml and virt-controller logs.

[shubham-pampattiwar]

Reproduction Results

Hi @ShellyKa13, we reproduced this with a fresh Fedora VM. Here are the artifacts you requested, along with new findings.

Environment:

• OpenShift CNV 4.99.0-0.1768264879
• Storage: ocs-storagecluster-ceph-rbd (default)
• Feature gates: IncrementalBackup, UtilityVolumes (via HCO annotation)
• CBT label selector: changedBlockTracking: "true"

Updated finding: No HotplugFailed on this run

Unlike the original report, on this reproduction the backup target PVC was correctly attached as a utilityVolume (type: Backup) — no HotplugFailed event occurred. However, the backup still fails because the QEMU backup job itself fails silently.

Root cause from virt-launcher logs

"Domain Job Completed event type 0 received. Job operation: 9, succeeded: false"
"Unexpected job completion type: 4 (only handling success case)"

The QEMU backup job failed (succeeded: false), but KubeVirt still marked the VMBackup as Done: True with reason "Successfully completed VirtualMachineBackup". The backup PVC is empty — no qcow2 files were ever written. The datamover pod then fails with no qcow2 files found in /backup-data.

There are also repeated errors in virt-launcher:

"Failed to get block info" reason="virError(Code=8, Domain=20, Message='string disk in virDomainGetBlockInfo must not be empty')"

This may be related — the utility volume in the VMI status shows target: "" (empty target name).

VM YAML (spec + status)

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: test-vm
  namespace: fedora-repro
  labels:
    app: test-vm
    changedBlockTracking: "true"
spec:
  running: true
  dataVolumeTemplates:
  - metadata:
      name: test-vm-rootdisk
      annotations:
        cdi.kubevirt.io/storage.deleteAfterCompletion: "false"
    spec:
      pvc:
        accessModes: [ReadWriteMany]
        resources:
          requests:
            storage: 30Gi
        storageClassName: ocs-storagecluster-ceph-rbd
        volumeMode: Block
      sourceRef:
        kind: DataSource
        name: fedora
        namespace: openshift-virtualization-os-images
  template:
    spec:
      architecture: amd64
      domain:
        devices:
          disks:
          - disk: {bus: virtio}
            name: rootdisk
          interfaces:
          - masquerade: {}
            name: default
          rng: {}
        features:
          smm: {enabled: true}
        firmware:
          bootloader:
            efi: {secureBoot: true}
        machine:
          type: pc-q35-rhel9.2.0
        resources:
          requests:
            memory: 2Gi
      networks:
      - name: default
        pod: {}
      volumes:
      - dataVolume: {name: test-vm-rootdisk}
        name: rootdisk
status:
  changedBlockTracking:
    state: Enabled
  printableStatus: Running
  ready: true
  conditions:
  - status: "True"
    type: Ready
  - status: "True"
    type: AgentConnected
  guestOSInfo:
    id: fedora
    name: Fedora Linux
    versionId: "43"
    kernelRelease: 6.17.1-300.fc43.x86_64

VMI status (key sections during backup)

spec:
  utilityVolumes:
  - claimName: kubevirt-backup-du-fedora-repro-backup-test-vm-b1ab8122-mfq4z
    name: vmb-du-fedora-repro-backup-test-vm-b1ab8vsn2p-backup-target-pvc
    type: Backup
status:
  changedBlockTracking:
    backupStatus:
      backupName: vmb-du-fedora-repro-backup-test-vm-b1ab8vsn2p
      checkpointName: vmb-du-fedora-repro-backup-test-vm-b1ab8vsn2p-2026-03-09_20-59-55
      startTimestamp: "2026-03-09T20:59:55Z"
    state: Enabled
  volumeStatus:
  - name: rootdisk
    persistentVolumeClaimInfo:
      volumeMode: Block
      claimName: test-vm-rootdisk
    target: vda
  - hotplugVolume:
      attachPodName: hp-volume-7csvl
      attachPodUID: 3409123b-eae6-4e66-93d9-3afe05c6dd90
    message: Volume vmb-...-backup-target-pvc has been mounted in virt-launcher pod
    name: vmb-du-fedora-repro-backup-test-vm-b1ab8vsn2p-backup-target-pvc
    persistentVolumeClaimInfo:
      volumeMode: Filesystem
      capacity:
        storage: 10Gi
    phase: MountedToPod
    target: ""

VMBackup YAML (with status)

apiVersion: backup.kubevirt.io/v1alpha1
kind: VirtualMachineBackup
metadata:
  name: vmb-du-fedora-repro-backup-test-vm-b1ab8vsn2p
  namespace: fedora-repro
spec:
  forceFullBackup: true
  pvcName: kubevirt-backup-du-fedora-repro-backup-test-vm-b1ab8122-mfq4z
  source:
    apiGroup: backup.kubevirt.io
    kind: VirtualMachineBackupTracker
    name: vmbt-test-vm-kbhzn
status:
  checkpointName: vmb-du-fedora-repro-backup-test-vm-b1ab8vsn2p-2026-03-09_20-59-55
  conditions:
  - lastTransitionTime: "2026-03-09T21:01:40Z"
    reason: Successfully completed VirtualMachineBackup
    status: "False"
    type: Progressing
  - lastTransitionTime: "2026-03-09T21:01:40Z"
    reason: Successfully completed VirtualMachineBackup
    status: "True"
    type: Done
  type: Full

Key questions for KubeVirt team

1. Why does the QEMU backup job fail silently? The job reports succeeded: false with completion type 4, but the VMBackup status says "Successfully completed". Should the VMBackup controller check the actual job result?
2. Is target: "" for the backup utility volume expected? The virDomainGetBlockInfo error with an empty disk string might be related to this empty target.
3. Is there a way to get more detail on why the QEMU backup job failed? The virt-launcher log only says succeeded: false without further diagnostics.

The cluster is available for investigation, happy to provide access or gather additional logs/data.

[ShellyKa13]

Hi @shubham-pampattiwar

1. I wonder regarding the cnv image you are using how updated it is, we added failure handling about 2 weeks ago.
2. yes target: "" is expected with utilityvolumes since target is the device in the guest which utilityvolume is not mounting into the guest. The error virDomainGetBlockInfo was also fixed with a PR that was merged even earlier on the middle of January in upstream. So thats another reason why maybe its better you use newer cnv images for your cluster if its possible. It will both help with debug and might also fix some bugs that were fixed lately.
3. It will be valuable to add annotation to the vm that adds libvirt and qemu debug info:
   vm.Spec.Template.ObjectMeta.Annotations["kubevirt.io/libvirt-log-filters"] = "3:remote 4:event 3:util.json 3:util.object 3:util.dbus 3:util.netlink 3:node_device 3:rpc 3:access 1:*"
   not sure if it requires a restart after adding this annotation to apply it for the next backup process.