Virtual SAN and Data-At-Rest Encryption

There are many reasons you might want to encrypt your data at rest.  This post will not discuss those reasons, but simply offer options on how to do it with Virtual SAN.  It should be noted that VMware takes customer feedback seriously and several customers have been asking for encryption to be built-in to VSAN. This is being appropriately considered and that’s about all I can say about that for now. As of Dec 2016, this is available in the latest vSAN Beta.

So what can customers do right now to solve their Data-At-Rest Encryption requirements for data residing on a Virtual SAN datastore? Here are 3 options.

  1. VM Based Encryption
    1. Third Parties
    2. VM Encryption
  2. vSAN Encryption (Coming Soon)
  3. Self Encrypting Drives (SED’s)
  4. Controller Based Key Management + Self Encrypting Drives (SED’s)

NOTE: The vSAN team tested SED’s and determined that they do not provide an acceptable solution. So, the only supported Encryption options are software based through third parties, using VM Encryption, or the upcoming vSAN Encryption. 

VM Based Encryption

The beauty of a VM based encryption solution is it works to protect any VM and its data sitting on any storage. This solution approach will work with VM’s on Virtual SAN but also with VM’s on any other datastore backed by any storage system (e.g. EMC, NetApp, HP, Dell, etc.) via any connectivity (e.g. Fibre Channel, iSCSI, NFS). Here are some options:

VM Based Encryption

At VMworld 2016 in Las Vegas, VMware announced the upcoming vSAN Beta. For more details see:
Virtual SAN Beta – Register Today!
https://blogs.vmware.com/virtualblocks/2016/09/07/virtual-san-beta-register-today/

This vSAN Beta includes vSAN encryption targeted for a future release of vSphere. vSAN Encryption will satisfy data-at-rest encryption.

You might ask why vSAN Encryption would be necessary if vSphere has VM Encryption? Check out this
The difference between VM Encryption in vSphere 6.5 and vSAN encryption
http://www.yellow-bricks.com/2016/11/07/the-difference-between-vm-encryption-in-vsphere-6-5-and-vsan-encryption/

Self Encrypting Drives (SED’s)

Self Encrypting Drives (SED’s) are just what they say they are. When blocks are ingested into these drives, the firmware on the drives will encrypt the data and complete the write. When blocks are read back, the firmware decrypts the data and sends it back. This is valuable in the case that if a drive goes missing, the data on it is encrypted. Here are a few details on how they work:

  • The encryption key used in SEDs is called the Media Encryption Key (MEK)
  • Locking and unlocking a drive requires another key, called the Key Encryption Key (KEK)
  • If no KEK is set, the drive is always unlocked and appears not to be encrypting even though it is
  • If a KEK is set, the drive will power up locked until the correct KEK is given to the drive by the user.

When used with Virtual SAN, these drives simply work without Virtual SAN even knowing encryption is going on and all the things that hold true for standard drives would hold true when using these SEDs. Like standard drives, if a host fails, you could pull the SED’s and move to a different host or a new host in the same cluster and the data would be intact. Like standard drives, the SEDs could not be moved and read by hosts in a different cluster.

SED’s have minimal impact to I/O latency since the encryption work is distributed across all the drives. There are SED HDD’s and SED SSD’s offered by different manufacturers and some will be qualified and made available on the VMware Compatibility Guide (VCG) for Virtual SAN.

http://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan

Controller Based Key Management + Self Encrypting Drives (SED’s)

Some customers want to introduce encryption keys and a centralized way to manage the keys so that they have greater control over the Self Encrypting Drives.

LSI (Avago) has a technology built into their IO controller cards called SafeStore that has built in key generation software that can be used in conjunction with the SEDs.

LSI™ MegaRAID® SafeStore™ Software FAQ

https://www.lsi.com/downloads/Public/Advanced%20Software/LSI%20MegaRAID%20SafeStore%20Software/MR_SafeStore_FAQ_042110.pdf

SafeStore essentially locks the drive and it cannot be powered on or moved to a new or another host in the cluster without using the SafeStore key to unlock it first. There would be one LSI SafeStore key for every LSI controller. SafeStore keys can be managed by 3rd party key management software (e.g. RSA, SafeNet, etc.). If a host fails but the drives are still good simply pull the drives out and put them in a new host. On power up they will be locked, so you’ll need to unlock them.

LSI IO Controllers that adhere to the SafeStore standard will be publish on the VMware Compatibility Guide for Virtual SAN.

Summary

VMware has a few options for Data-At-Rest encryption with Virtual SAN. Each have their own pros and cons so customers can weigh the odds and make the appropriate decision.

Configuring LSI MegaRAID 9271CV-81 I/O Controller for VSAN

A few colleagues of mine recently worked with a customer deploying VMware Virtual SAN (VSAN) with Cisco UCS hosts using the LSI MegaRAID 9271CV-81 I/O Controller and documented the configuration choices below. Note, in general the VSAN guidance has been to disable all controller cache so these choices follow this theme. Also note that we are not LSI experts and would welcome feedback from others on their experience with other settings.

Change the default settings in the screenshot below to the following:

  • Access = RW
  • I/O = Direct
  • Read = Disable
  • Disk Cache = Disabled
  • Disable BGI = No
  • Default Write = Write Through

LSI MegaRAID BIOS Config 01

* All settings can be changed on the fly or using storcli for VMware:

http://www.lsi.com/downloads/Public/RAID%2520Controllers/RAID%2520Controllers%2520Common%2520Files/1.09.13_StorCLI.zip 

* User guide

http://www.lsi.com/downloads/Public/RAID%20Controllers/RAID%20Controllers%20Common%20Files/51530-00_Rev_L.zip

Other versions of the MegaRAID controller might have a screen that looks something like the one below:

LSI MegaRAID BIOS Config 02

Thanks to my colleagues Justin Beck and Jason Burroughs for documenting and sharing their experience.