vmdk-convert reports incorrect disk capacity

[santeri3700]

Describe the bug

vmdk-convert seems to report incorrect "capacity" and "used" sizes for a thin provisioned VMDK file.

Using ova-compose generated OVF/OVA may result in failed imports to vCenter due to the incorrect size for the "ovf:capacity" value retrieved with vmdk-convert.

Reproduction steps

• Create a new VM on VMware vCenter 8.0
  • Create a single thin provisioned 15GB root disk
  • Install some operating system to use up space
• Export/download the VMDK from vCenter
• Use ova-compose and example.yaml config to create a new OVF.

  ./ova-compose.py --input-file example.yaml --output-file example.ovf --format ovf --checksum-type sha256 --param rootdisk=example-disk-0.vmdk
  

• Inspect the OVF contents and see that the disk sizes are incorrect. See more details below for analyzing the VMDK.

Expected behavior

vmdk-convert should report the correct size for "capacity". This should be the virtual size of the hard disk and not the real VMDK file size (e.g. 15GB in the example below).

Additional context

VMDK information reported by vmdk-convert (incorrect):

$ vmdk-convert -i almalinux-9-generic-x86_64-disk-0.vmdk
{ "capacity": 717702144, "used": 717702144 }

VMDK information reported by qemu-img (correct):

$ du -sb ./almalinux-9-generic-x86_64-disk-0.vmdk
717702144	./almalinux-9-generic-x86_64-disk-0.vmdk

$ qemu-img info --output json ./almalinux-9-generic-x86_64-disk-0.vmdk | jq '.["virtual-size"]'
15728640000

$ qemu-img info --output json ./almalinux-9-generic-x86_64-disk-0.vmdk | jq '.["actual-size"]'
717705216

Invalid OVF contents generated by ova-compose:

<File ovf:href="almalinux-9-generic-x86_64-disk-0.vmdk" ovf:id="file0" ovf:size="717702144"/>

<Disk ovf:diskId="vmdisk0" ovf:capacity="717702144" ovf:capacityAllocationUnits="byte" ovf:fileRef="file0" ovf:populatedSize="717702144" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized"/>

Valid OVF contents generated by vCenter:
NOTE: vCenter doesn't seem to set the populatedSize value.

<File ovf:href="almalinux-9-generic-x86_64-disk-0.vmdk" ovf:id="file1" ovf:size="717702144"/>

<Disk ovf:capacity="15000" ovf:capacityAllocationUnits="byte * 2^20" ovf:diskId="vmdisk1" ovf:fileRef="file1" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized"/>

VMware vAPI logs from a failed import:

ERROR    | null             | transferService-http-abc123...-almalinux-9-generic-x86_64-disk-0.vmdk-upload | HttpClientEndpointImpl         | Transfer session abc123...: Error uploading to https://HOST/nfc/foo-bar-123.../disk-0.vmdk: Response: 500 Internal Server Error: Capacity of uploaded disk is larger than requested
INFO     | ...ID... | transferService-pool-7-thread-7 | TransferEndpointImpl           | Session abc123..., Item almalinux-9-generic-x86_64-disk-0.vmdk: Transfer done for endpoint HttpClientEndpointImpl, size = 717702144
INFO     | ...ID... | transferService-pool-7-thread-7 | TransferEndpointImpl           | Session abc123...: Transfer failed! item: almalinux-9-generic-x86_64-disk-0.vmdk; bytes transferred: 717702144; total bytes: 717702144; error: Invalid response code: 500
DEBUG    | ...ID... | transferService-pool-7-thread-7 | HttpServerEndpointImpl         | Served endpoint for abc123.../almalinux-9-generic-x86_64-disk-0.vmdk: Not ready
DEBUG    | ...ID... | transferService-pool-7-thread-7 | TransferItemImpl               | Session abc123...: no transfer scheduled due to error on target https://HOST/nfc/foo-bar-123.../disk-0.vmdk for item almalinux-9-generic-x86_64-disk-0.vmdk: Invalid response code: 500.
DEBUG    | ...ID... | transferService-pool-7-thread-7 | TransferItemImpl               | Session abc123..., Item almalinux-9-generic-x86_64-disk-0.vmdk: Transfer summary: rate=37.13 MB/s, size=717702144, time=19328 ms, status=ERROR, type=DISK, source=https://VCENTER:443/cls/data/abc123.../almalinux-9-generic-x86_64-disk-0.vmdk, target=https://HOST/nfc/foo-bar-123.../disk-0.vmdk

Workaround for ova-compose

qemu-img reports correct sizes but would introduce a new dependency for this project therefore making this a unviable solution.

Commit: e13a54ff037c0653ff122bcf17d7c9d47ad18756

--- ova-compose.py
+++ ova-compose.py
@@ -1072,8 +1072,12 @@
 
     @staticmethod
     def _disk_info(filename):
-        out = subprocess.check_output([VMDK_CONVERT, "-i", filename]).decode("UTF-8")
-        return json.loads(out)
+        json_data = subprocess.check_output(['qemu-img', 'info', '--output', 'json', filename]).decode("UTF-8")
+        disk_info = json.loads(json_data)
+        return {
+            'capacity': disk_info['virtual-size'],
+            'used': disk_info['actual-size']
+        }
 
 
     def to_xml(self):

[oliverkurth]

The VMDK produced in VC/ESXi is not in a streamable format, and open-vmdk only supports that.

vmdk-convert tries to open it as a streamable VMDK but fails, and then opens it as a raw image file. So therefore, it reports the wrong size.

But it's true that vmdk-convert should throw an error instead of going on with it.

[oliverkurth]

So for your use case, you should convert the VMDK to a streamable format using qemu first.

[santeri3700]

Thanks for the info! I was under the impression that my downloaded VMDKs were already in streamable format. Even qemu-img seems to think they are (unless I'm misunderstanding something) but (re)converting the VMDK with qemu-img fixes the reporting issue with vmdk-convert.

$ du -sb original.vmdk 
710812160	original.vmdk

$ qemu-img info --output json original.vmdk | jq '.["format-specific"]["data"]["create-type"]'
"streamOptimized"

$ vmdk-convert -i original.vmdk 
{ "capacity": 710812160, "used": 710812160 }

$ qemu-img convert -f vmdk -O vmdk -o subformat=streamOptimized original.vmdk target.vmdk

$ du -sb target.vmdk
678005760	target.vmdk

$ qemu-img info --output json target.vmdk | jq '.["format-specific"]["data"]["create-type"]'
"streamOptimized"

$ vmdk-convert -i target.vmdk
{ "capacity": 16097738752, "used": 1431896064 }

VMware doesn't seem to care about the VMDK as long as the OVF capacity value is correct so perhaps vmdk-convert can be improved to report the correct capacity with these kind of VMDKs as well. I might have a look at it later and see if my C skills are good enough.

Should I leave this issue open?

[oliverkurth]

Let's leave this open - vmdk-convert should report the correct capacity if feasible, or throw an error.

[smogm]

If the vmdk was created with qemu-img and compat6 the vmdk-convert -i test.vmdk takes forever.
Seems there is an endless loop in https://github.com/vmware/open-vmdk/blob/master/vmdk/mkdisk.c#L186.

[oliverkurth]

Can you give exact repro steps?

[oliverkurth]

@smogm any update? Can you give me steps to reproduce?

Check if this patch would fix it:

diff --git a/vmdk/sparse.c b/vmdk/sparse.c
index 346491a..591a8a3 100644
--- a/vmdk/sparse.c
+++ b/vmdk/sparse.c
@@ -946,7 +946,7 @@ SparseNextData(DiskInfo *self,
                 *end = grainNr * (sdi->diskHdr.grainSize * VMDK_SECTOR_SIZE);
                 return 0;
             }
-            *pos = grainNr * (sdi->diskHdr.grainSize * VMDK_SECTOR_SIZE) | skip;
+            *pos = grainNr * (sdi->diskHdr.grainSize * VMDK_SECTOR_SIZE) + skip;
             want = true;
         }
         skip = 0;