Podcast Fun!

In my role I have to drive a lot around New England. To pass the time I listen to a number of podcasts. Some of my favorites include:

Job Related:

Fun stuff:

But by far my favorite and the most entertaining is:

Virtually Speaking

I guess it’s partly because it focuses on storage for VMware environments, but, it’s also because Pete Flecha and John Nicholson are the right amount of funny, geek, and attitude all rolled into one.

A few weeks ago I had the chance to sit with John Nicholson and Duncan Epping to record some sound bits regarding customer experiences with vSAN in the field. I get to meet and work with a lot of remarkable customers up and down the eastern USA and over the last 3 years I’ve seen them accomplish great things with vSAN. You name an application or use case and it’s pretty likely its being done with vSAN. I was able to share a few stories as was Josh Fidel (@jcefidel) who’s doing great things with vSAN at customers in the Michigan, Ohio, Indiana, and Kentucky areas. He’s no SLOB and don’t let him fool you, he’s as smart as he is interesting. Check out what I mean by listening to this episode:

Virtually Speaking Podcast Episode 36: vSAN Use Cases





vSAN and Data-At-Rest Encryption – Rebooted (i.e. Part 2)


Encryption is here, now shipping with vSphere 6.5.

I first wrote about vSAN and Encryption here:

Virtual SAN and Data-At-Rest Encryption – https://livevirtually.net/2015/10/21/virtual-san-and-data-at-rest-encryption/

At the time, I knew what was coming but couldn’t say. Also, the vSAN team had plans that changed. So, let’s set the record straight.


  • Does not support Self Encrypting Drives (SEDs) with encryption enabled.
  • Does not support controller based encryption.
  • Supports 3rd party software based encryption solutions like HyTrust DataControl and Dell EMC Cloud Link.
  • Supports the VMware VM Encryption released with vSphere 6.5
  • Will support its own VMware vSAN Encryption in a future release.

At VMworld 2016 in Barcelona VMware announced vSphere 6.5 and with it, VM Encryption. In the past, VMware relied on 3rd party encryption solutions, but now, VMware has its own. For more details, check out:

What’s new in vSphere 6.5: Security – https://blogs.vmware.com/vsphere/2016/10/whats-new-in-vsphere-6-5-security.html

In this, Mike Foley briefly highlights a few advantages of VM Encryption. Stay tuned for more from him on this topic.

In addition to what Mike highlighted, VM encryption is implemented using VAIO Filters, can be enabled per VM object (vmdk), will encrypt VM data no matter what storage solution is implemented (e.g. object, file, block using vendors like VMware vSAN, Dell Technologies, NetApp, IBM, HDS, etc.), and satisfies data-in-flight and data-at-rest encryption. The solution does not require SED’s so it works with all the commodity HDD, SSD, PCIe, and NVMe devices and integrates with several third party Key Management solutions. Since VM Encryption is set via policy, that policy could extended across to public clouds like Cloud Foundation on IBM SoftLayer, VMware Cloud on AWS, VMware vCloud Air or to any vCloud Air Network partner. This is great because your VM’s could live in the cloud but you will own and control the encryption keys. And you can use different keys for different VM’s.

At VMworld 2016 in Las Vegas VMware announced the upcoming vSAN Beta. For more details see:

Virtual SAN Beta – Register Today! – https://blogs.vmware.com/virtualblocks/2016/09/07/virtual-san-beta-register-today/

This vSAN Beta includes vSAN encryption targeted for a future release of vSphere. vSAN Encryption will satisfy data-at-rest encryption. You might ask why vSAN Encryption would be necessary if vSphere has VM Encryption? I will say that you should always look to use VM Encryption first. The one downside to VM Encryption is that since the VM’s data is encrypted as soon as it leaves the VM and hits the ESXi kernel, each block is unique, so no matter what storage system that data goes to (e.g. VMware vSAN, Dell Technologies, NetApp, IBM, HDS, etc.) that block can’t be deduped or compressed. The benefit of vSAN encryption will be that the encryption will be done at the vSAN level. Data will be send to the vSAN cache and encrypted at that tier. When it is later destaged, it will be decrypted, deduped, compressed, and encrypted when its written to the capacity tier. This satisfies the data-at-rest encryption requirements but not data-in-flight. It does allow you to take advantage of the vSAN dedupe and compression data services and it’s one key for the entire vSAN datastore.

It should be noted that both solutions will require a 3rd party Key Management Server (KMS) and the same one can be used for both VM Encryption and vSAN Encryption. The KMS must support the Key Management Interoperability Protocol (KMIP) 1.1 standard. There are many that do and VMware has tested a lot of them. We’ll soon be publishing a list, but for now, check with your KMS vendor or your VMware SE for details.

VMware is all about customer choice. So, we offer a number of software based encryption options depending on your requirements.

It’s worth restating that VM Encryption should be the standard for software based encryption for VM’s. After reviewing vSAN Encryption, some may choose it instead to go with vSAN encryption if they want to take advantage of deduplication and compression. Duncan Epping provides a little more detail here:

The difference between VM Encryption in vSphere 6.5 and vSAN encryption – http://www.yellow-bricks.com/2016/11/07/the-difference-between-vm-encryption-in-vsphere-6-5-and-vsan-encryption/


In summary:

  1. Use VM Encryption for Hybrid vSAN clusters
  2. Use VM Encryption on All-Flash if storage efficiency (dedupe/compression) is not critical
  3. Wait for vSAN native software data at rest encryption if you must have dedupe/compression on All-Flash


Correlating vSAN versions with vSphere (vCenter & ESXi) Versions

I often get asked if a certain version of vSAN can be deployed on a different version of vSphere. The answer is no. vSAN is built into the vSphere version. That means vCenter needs to be upgraded to the correct version of vCenter and all the hosts in the cluster need to be upgraded to the correct version of ESXi in order to get the features of that version of vSAN. Lastly, vSAN formats each disk drive with an on-disk format, so to get the full features of a specific release, you may need to update the on-disk format.

Here’s basically how everything breaks down:

  • If you have vSphere 5.5 (vCenter Server 5.0 & ESXi 5.0) then you have vSAN 5.5.
  • If you have vSphere 6.0 (vCenter Server 6.0 & ESXi 6.0) then you have vSAN 6.0.
  • If you have vSphere 6.0 U1 (vCenter Server 6.0 Update 2 & ESXi 6.0 Update 1) then you have vSAN 6.1.
  • If you have vSphere 6.0 U2 (vCenter Server 6.0 Update 2 & ESXi 6.0 Update 2) then you have vSAN 6.2.
  • If you have vSphere 6.5 (vCenter Server 6.5 & ESXi 6.5) then you have vSAN 6.5.
  • If you have vSphere 6.5.0d (vCenter Server 6.5.0d & ESXi 6.5.0d) then you have vSAN 6.6.
  • If you have vSphere 6.5 Update 1 (vCenter Server 6.5 Update 1 & ESXi 6.5 Update 1) then you have vSAN 6.6.1.

Here’s a more detailed matrix:

Version Release




Installer Build Number vSAN Version vSAN

On-Disk Format

(Web Client)

ESXi 6.6.1 Patch 02 2017-12-19 7388607 N/A 6.6.1 5
ESXi 6.5 Express Patch 4 2017-10-05 6765664 N/A 6.6.1 5
ESXi 6.5 Update 1 2017-07-27 5969303 N/A 6.6.1 5
ESXi 6.5.0d 2017-04-18 5310538 N/A 6.6 5
ESXi 6.5. Express Patch 1a 2017-03-28 5224529 N/A 6.5 3
ESXi 6.5. Patch 01 2017-03-09 5146846 5146843 6.5 3
ESXi 6.5.0a 2017-02-02 4887370 N/A 6.5 3
ESXi 6.5 GA 2016-11-15 4564106 N/A 6.5 3
ESXi 6.0 Express Patch 7a 2017-03-28 5224934 N/A 6.2 3
ESXi 6.0 Update 3 2017-02-24 5050593 N/A 6.2 3
ESXi 6.0 Patch 4 2016-11-22 4600944 N/A 6.2 3
ESXi 6.0 Express Patch 7 2016-10-17 4510822 N/A 6.2 3
ESXi 6.0 Patch 3 2016-08-04 4192238 N/A 6.2 3
ESXi 6.0 Express Patch 6 2016-05-12 3825889 N/A 6.2 3
ESXi 6.0 Update 2 2016-03-16 3620759 N/A 6.2 3
ESXi 6.0 Express Patch 5 2016-02-23 3568940 N/A 6.1 2
ESXi 6.0 Update 1b 2016-01-07 3380124 N/A 6.1 2
ESXi 6.0 Express Patch 4 2015-11-25 3247720 N/A 6.1 2
ESXi 6.0 U1a (Express Patch 3) 2015-10-06 3073146 N/A 6.1 2
ESXi 6.0 U1 2015-09-10 3029758 N/A 6.1 2
ESXi 6.0.0b 2015-07-07 2809209 N/A 6.0 2
ESXi 6.0 Express Patch 2 2015-05-14 2715440 N/A 6.0 2
ESXi 6.0 Express Patch 1 2015-04-09 2615704 2615979 6.0 2
ESXi 6.0 GA 2015-03-12 2494585 N/A 6.0 2
ESXi 5.5 Patch 10 2016-12-20 4722766 4761836 5.5 1
ESXi 5.5 Patch 9 2016-09-15 4345813 4362114 5.5 1
ESXi 5.5 Patch 8 2016-08-04 4179633 N/A 5.5 1
ESXi 5.5 Express Patch 10 2016-02-22 3568722 N/A 5.5 1
ESXi 5.5 Express Patch 9 2016-01-04 3343343 N/A 5.5 1
ESXi 5.5 Update 3b 2015-12-08 3248547 N/A 5.5 1
ESXi 5.5 Update 3a 2015-10-06 3116895 N/A 5.5 1
ESXi 5.5 Update 3 2015-09-16 3029944 N/A 5.5 1
ESXi 5.5 Patch 5 re-release 2015-05-08 2718055 N/A 5.5 1
ESXi 5.5 Express Patch 7 2015-04-07 2638301 N/A 5.5 1
ESXi 5.5 Express Patch 6 2015-02-05 2456374 N/A 5.5 1
ESXi 5.5 Patch 4 2015-01-27 2403361 N/A 5.5 1
ESXi 5.5 Express Patch 5 2014-12-02 2302651 N/A 5.5 1
ESXi 5.5 Patch 3 2014-10-15 2143827 N/A 5.5 1
ESXi 5.5 Update 2 2014-09-09 2068190 N/A 5.5 1
ESXi 5.5 Patch 2 2014-07-01 1892794 N/A 5.5 1
ESXi 5.5 Express Patch 4 2014-06-11 1881737 N/A 5.5 1
ESXi 5.5 Update 1a 2014-04-19 1746018 N/A 5.5 1
ESXi 5.5 Express Patch 3 2014-04-19 1746974 N/A 5.5 1
ESXi 5.5 Update 1 2014-03-11 1623387 N/A 5.5 1
ESXi 5.5 Patch 1 2013-12-22 1474528 N/A 5.5 1
ESXi 5.5 GA 2013-09-22 1331820 N/A 5.5 1

As a reference, see:

Build numbers and versions of VMware vSAN (2150753) – This is a new KB post that went up on July 31, 2017 which provides the same information as above.

Build numbers and versions of VMware vCenter Server (2143838)

Build numbers and versions of VMware ESXi/ESX (2143832)

Understanding vSAN on-disk format versions (2145267)






How to Change All of the vSAN VMkernel Port IP Addresses in a vSphere Cluster.

Several months ago I was asked how to change all the vSAN VMkernel port IP Addresses in a vSphere cluster and today I was asked again, so here it is.


Assuming each host has 2 VMkernel ports (a & b) enabled for vSAN traffic.

  • Disable vSAN traffic on each of the b networking interfaces on each host
  • Change the IP addresses on each of the b networking interfaces on each host
  • Move the physical network cable if moving to new switch ports
  • Re-enable vSAN traffic on each of the b networking interfaces on each host
  • Verify communication between all the b networking interfaces using vmkping test.
  • Repeat for all the a networking interfaces

Disruptively (downtime is OK and/or the hosts are being moved)


vSAN,… correction its VSAN,… OK, OK, its vSAN now, VSAN and Virtual SAN are wrong.

I spent 4 years at EMC prior to moving to VMware over 3 years ago to join the Software Defined Storage team. At EMC it was always a challenge to get the acronyms and names correct. When Acadia (VCE) first came out with the “Vblock” everybody wanted to type it as “vBlock”. I’d always try to subtly correct it and hope people got the hint. Other times I’d straight out correct them and feel like a jerk. But, to me, using the proper name was and is important. The same problem happened with “VPLEX”, everyone wanted to type it “vPLEX”. Why did people want to do this? Well, it’s VMware’s fault because they named things like “vSphere” and “vCenter” and later “vCloud” and “vRealize”. So when I joined VMware it was odd to me that we called our upcoming product “VSAN” and not “vSAN”. I’ve spent 3 years correcting Customers and VMware people one way or another that publically, and in product documentation, VMware actually could only call our product “Virtual SAN”. Many people, including me, got lazy and called it “VSAN”… but it was definitely not “vSAN”. Well, yesterday that all changed. Without going into detail, “vSAN” is the only name to use. Virtual SAN and VSAN are no more. Now I have to go fix all my spell checkers.

VMware Storage Technology Names & Acronyms

  • vSAN = VMware’s Software Defined Storage Solution formerly known as Virtual SAN or VSAN. Now the only acceptible name is “vSAN” with the little “v”.
  • SPBM = Storage Policy Based Management
  • VASA = vSphere API’s for Storage Awareness
  • VVol = Virtual Volume
  • PE = Protocol Endpoint
  • VAAI = vSphere API’s for Array Integration
  • VAIO Filtering = vSphere API’s for IO Filtering
  • VR = vSphere Replication
  • SRM = Site Recovery Manager
  • VDP = vSphere Data Protection
  • vFRC = vSphere Flash Read Cache
  • VSA = vSphere Storage Appliance (end of life)
  • VMFS = Virtual Machine File System
  • SvMotion = Storage vMotion
  • XvMotion – Across Host, Cluster, vCenter vMotion (without shared storage)
  • SDRS = Storage Distributed Resource Scheduler
  • SIOC = Storage Input Output Control
  • MPIO = Multi Path Input Output

Citrix & VSAN

There are many VMware and Citrix customers happily running Citrix XenApp and XenDesktop on VMware vSphere clusters with Virtual SAN enabled.

Citrix XenApp is fully supported on VSAN.

Citrix XenDesktop PVS is fully supported on VSAN.

Citrix XenDesktop MCS is still not supported on VSAN by Citrix at the time of this writing on October 7, 2016. Citrix has a fix that is in 7.8 and 7.9 already and customers have reported that the fix works, however Citrix claims the fix has not been qualified by them and thus is not supported. ETA for their official support is unclear at this point but is the responsibility of Citrix. If you are needing this feature, please reach out to Citrix to let them know.

Our friends at Dell Technologies (EMC/VCE) have tested XenApp, XenDesktop PVS and MCS on VxRail and have produced a report here:

Citrix XenDesktop 7.9 and VMware vSphere 6.0 with VCE VxRail Appliance

In it they state “Citrix official support of MCS on VMware Virtual SAN is expected in a future release of XenDesktop. EMC tested this configuration and found no observable issues.

For the record, I’ve been a fan of Citrix since I first deployed Citrix WinView in my data center and remote sites back in 1994. Yes, I’m that old. I’m sure this will all get worked out.