Live Virtually

I cannot take credit for this post, I posed the question regarding “Best Practices for locating the Virtual Machine swap file and several vSpecialist responded with number of great answers / opinions / recommended best practices for that and for Guest OS Paging. So thanks to Robert Way, Rajeev Dawar, Rich Barlow, Itzik Reich, Scott Drummonds, and Erik Zandboer for their contributions.

Here’s one of the best general reads on this topic.  “(Alternative) VM swap file locations Q&A” http://frankdenneman.nl/vmware/alternative-vm-swap-file-locations-qa/

The general consensus is that if you are swapping you are screwed.  Once swapping starts, very bad things are already happening, so a design that is centered around identifying and stopping swap as soon as it happens is best.  Sure, EMC would love it if you put your swap files on EMC Flash technology to reduce the performance impact of swapping.  But this might not be the best use of Flash resources if you are concerned about cost, and who isn’t?  If you’ve done a good job architecting your VM’s then you shouldn’t be overcommitting memory and thus it shouldn’t really matter where you put the swap files.  So in general you could keep things simple and just accept the default “Virtual machine directory – Store the swap files in the same directory as the virtual machines”.  With EMC’s FAST VP technology on the VNX and VMAX, if the swap files are not used then they will be migrated to the lowest tier.

There are several good reasons to choose the option: “Datastore Specified by host – Store the swap files in the datastore specified by the host to be used for swap files.  If not possible, store the swap files in the same directory as the virtual machine.”

• Capacity –  The VM swap file space can add up for a large number of VM’s so isolating them to a separate datastore can provide some efficiencies.
• Replication – The VM swap file should be in a non-replicated datastore.  If swapping is occurring then the blocks are constantly changing thus it will be constantly replicated possibly resulting in a very busy WAN link resulting in poor performance and a nightmare for VMware Site Recovery Manager (SRM).
• Snapshot – Similar to replication, the VM swap file swap should not be on a datastore with snapshots enabled.  This will save on storage capacity and provide better overall efficiency.
• Backup – Similar to replication, the VM swap file should be on a datastore that is not backed up in order to skip scanning and backing up swap file data.   Here’s a good reference: “What Do You Do With Your Swap File? – VMworld Update, Thursday” http://thebackupwindow.emc.com/alex_almeida/what-do-you-do-with-your-swap-file-vmworld-update-thursday/
• Troubleshooting – isolation helps in finding guest OS issues but resolving the underlying overcommit issues should be the focus.  Here’s a good reference: “Storing a virtual machine swap file in a location other than the default” http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004082

And a reason to keep the VM swap files on a shared datastore versus a local datastore:

• vMotion Performance – If the destination host cannot see the swap file then it creates a new swap file and potentially copies swapped memory from the source.  This can increase the time it takes to perform a vMotion but might be a worthy tradeoff to leveraging the host’s local disk.

If you have a good number of virtual machines then based on what’s written above, there are more compelling reasons to choose to put your VM swap files in a different location.  The next question is “Where do you put them?”  If you agree that if your swapping you’re screwed anyway then it makes sense to just put them somewhere cheap and forget about it.  Here is some guidance related to EMC storage.

Option 1 – Put them somewhere cheap and monitor closely for swapping

Create a LUN or volume on the least expensive storage tier.  This could be a dedicated NL-SAS or SATA disk group or with EMC FAST VP technology you could force the LUN to the lowest tier.  Connect all the hosts in the cluster to this datastore and use it as a consolidated location for all the VM swap files in the cluster.  Since the performance will be horrible if swapping occurs, the IO’s to this datastore should be closely monitored and if the alarm triggers then spring into action and go stop the swapping.

Option 2 – Let FAST VP do its thing

Create a LUN or volume in a FAST VP pool.  With EMC’s FAST VP technology on the VNX and VMAX, if the swap files are not used then they will be migrated to the lowest tier, thus the most cost effective tier for swap files.  You could also force the LUN or Volume to the lowest tier upon creation.  Connect all the hosts in the cluster to this datastore and use it as a consolidated location for all the VM swap files in the cluster.  Then, as in Option 1, monitor the IO’s.

Option 3 – Plan for good performance if swapping occurs

There may be some applications where you do not want to risk poor performance if swapping occurs.  Or, there may be applications that heavily utilize the available swap space.  One such application is SAP.  This is where Flash technology can dramatically help.  There are a few choices here:

1. Create a LUN or volume in a FAST VP pool on a VNX or VMAX that included EFD’s and force the LUN or Volume to the highest tier upon creation.  Connect all the hosts in the cluster to this datastore and use it as a consolidated location for all the VM swap files in the cluster that you care about swap performance.
2. Install a VF Cache card on the host and dedicate a portion of it as local storage.  Then create a datastore using this local storage and point the VM swap files to it.  The swap file performance will be great but the vMotion performance will be impacted since other hosts cannot see this storage.
3. Leverage local EFD’s on the server to create a datastore and point the VM swap files to it.  The swap file performance will be good but the vMotion performance will be impacted since other hosts cannot see this storage.
4. Create a LUN or volume on EMC’s “Project X” (XtremIO) all-cache-array and point the VM swap files to it.  Other factor to consider is if you are starting a large number of VM’s in a short time (E.g. VDI) then the creation of the VM swap files can cause an impact to performance.  This should be considered in deciding the location of VM swap files.  “Project X” shines in this use case but is not available today so stay tuned in 2013.

It’s important to note that the above discussion was around ESX host swapping.  Don’t forget about the other swapping: in the guest OS.  For this we used the term “paging” to differentiate it from ESX swapping.  If you really want to fully manage paging you should create a virtual disk for each guest OS’s page file on the same datastore as the VM swap file.  Then if you are monitoring for IO’s to that datastore, the storage alarm will detect host swapping or guest paging.

Summary

To keep things simple you can leave the VM swap file and OS page file in the default locations, place the VM in cluster with a shared datastore backed by a LUN or volume in a FAST VP pool, and monitor IO’s closely.

If you want to gain maximum efficiency then choose a location other than the default for the VM swap file and OS page file, place them on a datastore that is backed by the lowest tier of storage (e.g. NL-SAS RAID group or a FAST VM pool with NL-SAS), and monitor IO’s closely.

If you are concerned with performance if swapping occurs then choose a location other than the default for the VM swap file and OS page file and place them on a datastore that is backed by the Flash technology either on the array or host.

As always, questions, comments, feedback is welcome.