VMware ESXi 4.1 does not use whole disk capacity for VMFS3

There seems to be a problem with the new version of VMware's ESXi, 4.1. Due to ESXi's automatic installation and disk partitioning, no advanced parameters can be given, e.g. to manually create vmfs3 partitions. Usually that works fine, the installation creates the system partitions (HyperVisor) and uses the rest of the disk for a local vmfs3 datastore.

The situation: I installed ESXi 4.1 on a Dell PE 2900 with 8 harddisks on a Raid-5, a total of 2.86 TB of diskspace. The integrated Raid Controller (PERC 5/i) shows the correct sum of diskspace and the installation of ESXi detects the logical harddisk correctly with a diskspace of 2.86 TB.

The problem: Once ESXi 4.1 was installed, a local VMFS datastore was created - with a size of 744 GB. Instead of using the whole diskspace. The maximum filesystem limit of VMFS3 is 2TB per LUN and ESXi detects the local disk/partition as a LUN so it should have created a 2TB vmfs datastore. But no.

Reinstallation
Maybe this problem was a coincidence so a complete reinstall could maybe solve the problem. But even after running the ESXi installer again, the same size of vmfs datastore was there: 744 GB.

Update September 3 2010 (Installation of another ESXi)
It seems that this whole problem only affects servers with a total disk capacity of over 2TB. As I describe in today's blog post the vmfs3 partition was created fine with a capacity of 1.4TB.

Manual repartitioning
Before losing too many words, I let you know from the begin: THIS WORKS. You can continue to read on, hopefully with big eyes and a smile on your face...

Now to the funny part! The following instructions are advanced system commands. Do not do them on a production machine, only on a newly installed ESXi 4.1 machine which doesn't host any virtual machines yet.

First we have to find the correct disk, for which ESXi has given a special name. Use the following commands to find your local disk. Note that I have cut the full output, I only show the local disk information (the full output would also contain CD drive, attached iSCSI or SAN disks, etc.).
To run the following commands, you need to enable SSH. You can do this on the ESXi console in Troubleshooting.

This shows the disk name (naa.6001e4f01c94d50013d852397c7ef00d) and the LUN name (vmhba1:C2:T0:L0):

# esxcfg-mpath -b
naa.6001e4f01c94d50013d852397c7ef00d : Local DELL Disk (naa.6001e4f01c94d50013d852397c7ef00d)
   vmhba1:C2:T0:L0 LUN:0 state:active Local HBA vmhba1 channel 2 target 0

With the following command we see even more information and what we need is also here: The device path which we will use as the local disk identifier in the next commands:

# esxcfg-scsidevs -l
naa.6001e4f01c94d50013d852397c7ef00d
   Device Type: Direct-Access    Size: 3000704 MB    Display Name: Local DELL Disk (naa.6001e4f01c94d50013d852397c7ef00d)    Multipath Plugin: NMP    Console Device: /vmfs/devices/disks/naa.6001e4f01c94d50013d852397c7ef00d    Devfs Path: /vmfs/devices/disks/naa.6001e4f01c94d50013d852397c7ef00d    Vendor: DELL Model: PERC 5/i Revis: 1.03    SCSI Level: 5 Is Pseudo: false Status: on    Is RDM Capable: false Is Removable: false    Is Local: true    Other Names:      vml.02000000006001e4f01c94d50013d852397c7ef00d504552432035    VAAI Status: unknown

Disk /vmfs/devices/disks/naa.6001e4f01c94d50013d852397c7ef00d: 3146.4 GB, 3146466197504 bytes
64 heads, 32 sectors/track, 3000704 cylinders, total 6145441792 sectors
Units = sectors of 1 * 512 = 512 bytes

Check your current partition table and note down the number of partition which is used for VMFS (by default it should be p3):

# fdisk -l

The next step is to delete the automatically created vmfs3 partition using the fdisk command:

fdisk -u /vmfs/devices/disks/naa.6001e4f01c94d50013d852397c7ef00d
Command (m for help): d
Partition number (1-8): 3
Command (m for help): w

Now we create a new partition and change its type to VMFS. When fdisk asks for the last sector (=size) of the new partition, we enter +2097152M (which is 2TB):

fdisk -u /vmfs/devices/disks/naa.6001e4f01c94d50013d852397c7ef00d
Command (m for help): n
Command action
   l logical (5 or over)
   p primary partition (1-4)
p
Selected partition 3
First sector (10229760-1850474495, default 10229760): 10229760
Last sector or +size or +sizeM or +sizeK (10229760-4294967294, default 4294967294): +2097152M

Command (m for help): t
Partition number (1-8): 3
Hex code (type L to list codes): fb
Changed system type of partition 3 to fb (VMFS)

Command (m for help): w

Now we check again the partition table to verify the changes:

# fdisk -l
/dev/disks/naa.6001e4f01c94d50013d852397c7ef00dp1     5 900 917504 5 Extended /dev/disks/naa.6001e4f01c94d50013d852397c7ef00dp2     901 4995 4193280 6 FAT16 /dev/disks/naa.6001e4f01c94d50013d852397c7ef00dp3     4996 2004996 2047999936+ fb VMFS /dev/disks/naa.6001e4f01c94d50013d852397c7ef00dp4 *   1 4 4080 4 FAT16 <32M /dev/disks/naa.6001e4f01c94d50013d852397c7ef00dp5     5 254 255984 6 FAT16 /dev/disks/naa.6001e4f01c94d50013d852397c7ef00dp6     255 504 255984 6 FAT16 /dev/disks/naa.6001e4f01c94d50013d852397c7ef00dp7     505 614 112624 fc VMKcore /dev/disks/naa.6001e4f01c94d50013d852397c7ef00dp8     615 900 292848 6 FAT16

Now the new partition has to be formatted to a VMFS3. This can be done with the following command where -b is standing for the filesystem blocksize. Here I use 8M which is currently the biggest blocksize and made for big vmdk files. Note that the partition number has to be given, therefore the :3 at the end:

# vmkfstools -C vmfs3 -b 8M -S datastore1 /dev/disks/naa.6001e4f01c94d50013d852397c7ef00d:3
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Creating vmfs3 file system on "naa.6001e4f01c94d50013d852397c7ef00d:3" with blockSize 8388608 and volume label "datastore1".
Successfully created new volume: 4c45bc40-6aa5a458-e509-001e4f2a6fac

Congratulations, the new VMFS datastore with a size of 2TB has been created on your ESXi 4.1 machine.

Unfortunately this still doesn't use the whole disk space of 2.86 TB but I figured this out too, however I do not recommend to do it (it could cause system instability) and you should be an advanced systems engineer to do it...
Update Oct 22 2010: As of now the ESXi server in question is still running with this manual partition, there have no instabilities been detected and ESXi is running just fine. However, I don't know what would happen if a patch has to be installed or an upgrade to a newer version (future releases) is pushed. Due to the manual partitioning I just don't know what the ESXi patch/upgrade setup would do.

If you really want to use the whole disk space, do the following tasks:

• Boot with your favorite Rescue CD and launch a partitioning tool (I used Knoppix 6.2 and started Gparted).
• Copy the data of the partition /dev/sda2 (the one with size 4GB and which is FAT16-formatted) to another partition. You will find three folders in this partition: downloads, log, var.
• Then delete /dev/sda2 in Gparted.
• Expand/Resize the extended partition up to a size of 1.5TB (you can try to use the full disk space, but it won't work due to a maximum extended partition size).
• Create a new logical partition within the extended partition with a size of 4GB and format it FAT16 - so we recreate the partition which we deleted before.
• Restore the files you copied to another partition before back to the new partition.
• Now create another new logical partition within the extended partition and use the rest of the extended partition space, do not format it (unformatted) - this will be the first vmfs partition.
• Create a new primary partition and use the rest of the harddisk, do not format it (unformatted) - this is our second vmfs partition.
• Once all this is done, restart your machine and boot ESXi.
• Once logged in back with SSH, you have to format the two new created partitions with the vmkfstools command (see above). Use fdisk -l to see the new partitions.