There is a bug in the way the object buffer cache size is calculated for vMotion. This bug only appears with Hardware version 8 virtual desktops with large screen resolutions, however only if 3D is disabled.
When 3D is enabled ESxi5 takes the necessary screen objects into account in the calculations and allocate sufficient buffer for vMotion to succeed. But with 3D disabled it doesn’t calculate the values correctly for the buffers. This is will impact any VM with large screen size.
If you have a peek at my previous post you will be able to identify the memory and swap differences between resolutions and number of displays. VMware View 5.0 Video RAM Storage sizing.
There are few workarounds and VMware has already published a KB for the error here.
My personal choice would would be to change add the parameter to the /etc/vmware/config file on the target hosts. This adds the option to every VMX process that is spawning on the host, which happens when vMotion is starting a virtual machine on the server. This approach will save you from having to edit all of the VM’s.
migrate.baseCptCacheSize = "16777216"
* Note that the number quoted should be the amount of video RAM to be assigned to the virtual desktops.
The above migrate.baseCptCacheSize parameter sets the base cache size to 16MB instead of the default which is 5MB. The above setting is a VMX setting but by adding it to /etc/vmware/config it gets applied to every VM on the host.
Another option is to enable 3D, however this will add an overhead memory reservation by 256MB. Read more at the VMware KB here.