Mike Hays - Virtualization, PowerShell, and more...

Here’s the scenario: You need to move a large number of ESX guest OSs from one datacenter to another.  To make things interesting, there isn’t enough bandwidth on the WAN to move the guest OSs on the wire or replicate the data, so it has to be out of band.  Also, you’re on a tight budget and timeline, so you have to use large commodity USB, Firewire, or eSATA drives to move the guest OSs.  Also, the source ESX servers are a mixture of ESX 2.x and 3.x.

That’s a lot to contend with, but here is one way to do it with limited downtime (1-3 days depending on the data footprint and shipping time).

1. Install vCenter Converter 4.0 Server/Client Standalone on a system that has the required external drive connection (USB, eSATA, etc.).  If you’re using USB, make sure it is USB 2.0.  Even if you know the system is USB 2.0, make sure it is enabled in the system BIOS (HP disables USB 2.0 for some reason on many of their server models that support USB 2.0.)
2. Create a local administrator account in each Windows guest, and make sure you know the root password for any *nix guests.
3. Note the amount of RAM and number of CPUs allocated for each guest.  You’ll need this later when you import the guest, as Converter will probably “truncate” these resources to make the guest “fit” in a running state on your Converter server system.  (<rant>This is annoying and Converter should warn you that it is going to do this, and give you the option to not change the guest OS configuration since it is common to not run the guest on the Converter server itself.</rant>).
4. Optional: Pre-configure the guest with the destination IP address and associated information (gateway, etc.).
5. Shutdown the guest OS.
6. Use Converter to export the guest OS to the USB drive.  Note that you must connect to a VirtualCenter 2.x server if the guest is hosted on ESX 2.x.  I had better luck when I connected directly to ESX 3.x instead of VirtualCenter which timed out periodically.  This might be environmental and your mileage may vary.  Either way, I recommend you tell Converter to export to Workstation 5.0 format, and that you tell it in the second to last wizard page not to modify the guest OS.
7. Repeat steps 2-5 above as needed.
8. Transport the drives to the destination.
9. Reverse the process above, using Converter to import a Workstation guest OS and “converting” it to your ESX server or vCenter server.  Feel free to modify the guest OS by installing the VMware tools or changing it’s memory/CPU allocation as needed.  You will probably need to adjust the number of CPUs and/or RAM in the guest based off what you documented in step 3 above.

Note that Converter only pulls the actually data out of the vmdk file.  For example, if you have a 200 GB vmdk  file with only 4 GB of data, Converter will only pull 4 GB of data across the network, and only use 4 GB of space on your external storage device.  Even so, you will likely see transfer rates of only 0.5GB-1GB/minute, so expect several hours to days of downtime if you are moving guests with tens to hundreds of gigabytes of data.

I tried several ways to find and eliminate any bottlenecks and increase Converter’s throughput.  But the throughput was about the same whether I used an external USB 2.0 disk or local internal disk, whether I used 100Mb/Full or 1Gb Ethernet, or whether I used a system with multiple CPUs/cores or not (although an old laptop I tried initially did cause performance issues).  The bottleneck appears to be in Converter itself and I’ve read of others who experienced similar throughput using it.

Approaches that might improve throughput, but create other issues:

Use vmkfstools: Using vmkfstools is faster, but copying the exported disks onto the USB drive erases most of the time saved.  You can connect the USB drive directly to an ESX server, but the Service Console is rather slow when accessing USB, even USB 2.0.  Also, since vmkfstools is a vmdk disk tool, you are exporting the disks and not the guest itself.  So you have an vmx file you need to copy as well.  This might be acceptable, and if all your disks are on the same volume and you’re running ESX 3.x you can even get away with copying the vmx file unmodified.  But if your disks are on different LUNs, or your ESX server is 2.x, you might find yourself spending more time in vmx files than any sane person should.  I’m all about scripting, and you can definitely script changes to vmx files, but depending on the environment, there can be a large number of variables for which you have to account.

Naturally you can recreate the vmx file from scratch using your favorite approach to creating a new guest OS.  But this can be time consuming if you are moving a large number of guests.

Use cp: This is generally a bad idea.  The ‘cp’ command takes 1MB nibbles of vmdk files which generates a large number of LUN locks.  Also, at least on ESX 2.x, using ‘cp’ to copy a vmdk file to a non-VMFS partition would corrupt the vmdk file.

Install Converter server in the ESX Service Console (temporarily): It can be done and reducing the number of hops seems like a good idea if you don’t mind rebuilding an ESX server when your done, but when I attempted this approach it actually doubled my export times.  As I stated above, the Service Console reportedly doesn’t handle USB 2.0 very well, but I think Service Console memory was actually my bottleneck, and you can’t give the Service Console more than 800 MB of RAM.  If you attempt this approach, note that you will need to change the HTTP and HTTPS TCP ports during the Converter server installation, and you will need to use a remote Converter client.

Use an NFS device you can transport: Now you’re talking.  If you are using ESX 3.x and have an NFS device with enough disk space that you can transport, you could use this option, and it would be faster.  Instead of using Converter, you can either clone or cold migrate the guests.  And if the NFS device can perform well enough, you could even reduce downtime by using storage migration to move running guests to the NFS device (Note: As of this writing VMware doesn’t support hot storage migrations where NFS is concerned, but it reportedly works fine in most cases).  Naturally you will need an NFS device you can move, and this device may require special consideration when shipping.

Use a SAN you can transport: This too would be faster and will even work with ESX 2.x, but you’ll need an extra SAN, and I didn’t have one of these laying around.

Replicate the data: Now your breaking the rules. This is probably the ideal solution, but it’s not out-of-band.  You can replicate the LUNs, make them visible to your ESX server, register the guests, and go.  VMFS2 LUNs will need an upgrade to VMFS3 if you are exclusively using ESX 3.x at your destination.  You can also use vmkfstools to export the guests vmdk files and copy them over the WAN, importing them on the other end.  Note that VMware doesn’t support using Converter over a WAN–expect timeouts if you attempt to do so.

Rebuild the guest OS from scratch: In some cases this is actually a good option.  For example, if there is network connectivity between the sites (just not enough to replicate gigabytes of data), and you have a Windows Active Directory domain controller you need to “move” it is probably cleaner to build the server fresh at the destination.  This might apply to other infrastructure servers such as DHCP, WINS, or DNS servers as well.

The process above is the best I’ve been able to find for out-of-band relocations of guest OSs.  Please give any feedback if you have found any better approaches that meet the requirements of the scenario or have other ideas to make this type of migration more pleasant.

Good luck, and happy virtualizing!