How to delete the RAID configuration from drives managed by the Cisco 12G SAS Modular Raid Controller

The content of this blog post was created by a couple of colleagues of mine, David Boone and Bradford Garvey. These guys do a phenomenal job making sure VMware Virtual SAN customers get a great experience with the product by helping them plan, configure, and test VSAN. Because of this, they end up uncovering interesting information like what’s to follow.

Sometimes Cisco UCS hardware intended to be used for VMware Virtual SAN has previously been configured for other uses. In these cases, sometimes a RAID configuration has already been configured on the drives. VSAN requires the individual drives be presented to ESXi either raw via the I/O controller set in Pass-Through Mode (See: How To Configure the Cisco 12G SAS Modular Raid Controller for Pass-Through Mode) or disks set in their own RAID 0 disk groups. Best practice is to set the I/O Controller in Pass-Through Mode (Enable JBOD).  However, if a RAID configuration previously existed, on the Cisco UCS platform there are a few extra steps to complete after enabling JBOD mode for the controller.

If drives were already configured as RAID virtual devices, delete the RAID configuration from the drives.  One way to do that is to Clear the entire VD configuration:

Clear the entire VD configuration

  • Log into the Cisco UCS Manager
  • Open a console to the host
  • Reboot the host
  • On boot up hit Ctrl+R to enter the Cisco 12G SAS Modular Raid Controller BIOS Configuration Utility
  • Hit Ctrl-N until the “VD Mgmt” page is selected
  • In the “VD Mgmt” screen, navigate to the controller, and press the F2 key.
  • Navigate to “Clear Configuration” and press Enter.  You should see this popup:

CiscoUCS - Remove RAID 1

  • Press “Yes” to delete all the virtual drives

Drives will then be in an “Unconfigured Good” state.  They might look something like this:

CiscoUCS - Remove RAID 2

If you see this, these 10 drives are in an “Unconfigured Good” state. They need to be converted to a JBOD state.

There are two options. You can convert a bunch of Unconfigured Good drives to JBOD drives (from the “VD Mgmt” screen) or you can convert a particular Unconfigured Good drive to a JBOD drive (from the “Drive Management” screen)

Option 1: Convert a bunch of Unconfigured Good drives to JBOD drives

Perform the following steps to convert a bunch of Unconfigured Good drives to JBOD drives:

  • In the “VD Mgmt” screen, navigate to the controller and press the F2 key.
  • Navigate to “Make JBOD”, and press Enter.
    The “Convert Unconfigured Good to JBOD” dialog appears, which shows all Unconfigured Good drives in the system.

CiscoUCS - Remove RAID 3

 

  • Select the Unconfigured Good drives which you want configured as JBODs for VSAN.
    To select or deselect all the Unconfigured Good drives at one go, select the topmost square backets in the “Unconfig good drives” box.
  • Press “OK”.
    The selected Unconfigured Good drives are converted to JBOD drives.

Option 2: Convert a particular Unconfigured Good drive to a JBOD drive

Perform the following steps to convert a particular Unconfigured Good drive to a JBOD drive:

  • In the “Drive Management” screen, navigate to an Unconfigured Good drive, and press the F2 key.
  • Navigate to “Make JBOD”, and press Enter.
  • Press “OK” in the message confirmation box to continue.

After converting all the 10 drives above to JBOD, the screen looks like this:

CiscoUCS - Remove RAID 4

Result

After rebooting, the BIOS will report all 10 drives and ESXi will see all of them in a JBOD (Pass-Through) configuration, with all the benefits of JBOD like being able to retrieve S.M.A.R.T.S. info from the physical drives.

The information obtained to create this post was gathered from the Avago – 12Gb/s MegaRAID® SAS Software – User Guide

Thanks again to David Boone and Bradford Garvey for providing this information.

How To Configure the Cisco 12G SAS Modular Raid Controller for Pass-Through Mode

Yesterday I was at the New England VTUG event which is always a great event to meet up with familiar faces and be introduced to some new ones. I met up with a relatively new VMware Virtual SAN customer and we discussed lots of fun things about VSAN and their implementation experience. One frustrating thing they mentioned is that they couldn’t find anywhere that documented how to put the Cisco 12G SAS Modular Raid Controller in Pass-Through mode. They explained that after lots of searching on VMware and Cisco’s site, they contacted Cisco and were provided the information. They were kind enough to capture a screenshot of the setting and provide it to me.

The procedure is:

  • Log into the Cisco UCS Manager
  • Open a console to the host
  • Reboot the host
  • On boot up hit Ctrl+R to enter the Cisco 12G SAS Modular Raid Controller BIOS Configuration Utility
  • Hit Ctrl-N until the “Ctrl Mgmt” page is selected
  • In the bottom right hand corner, make sure the “Enable JBOD” field shows an X per the screen shot below.
  • Hit Ctrl-S to save Reboot

Cisco 12G SAS Enable JBOD

That’s it. Easy.

If this is a brand new, unconfigured host, the unclaimed disks in the host will now get passed to ESXi and VSAN can use them for the VSAN datastore.

However, if this host IO Controller had previously been configured with RAID, you should check out: How to delete the RAID configuration from drives managed by the Cisco 12G SAS Modular Raid Controller

I hope that helps others save some time in getting VSAN up and running.

Special thanks to Stephanie Forde and Matthew Gabrick from the Boston Water and Sewer Commission for pointing this out and providing the screenshot.

Queue Depth and the FBWC Controller Cache module on the Cisco 12G SAS Modular Raid Controller for Virtual SAN

If you scan the bill of materials for the various Cisco UCS VSAN ReadyNodes you’ll see a line item for:

Controller Cache: Cisco 12Gbps SAS 1GB FBWC Cache module (Raid 0/1/5/6)

If you’ve followed Virtual SAN for awhile you might wonder, why would the ReadyNodes include controller cache when VMware recommends disabling controller cache when implementing Virtual SAN. Well, it turns out that the presence of the FBWC Cache module allows the queue depth of the Cisco 12G SAS Modular Raid Controller to go from the low 200’s to the advertised 895. The minimum queue depth requirement for Virtual SAN is 256 so including the FBWC Cache module allows the queue depth to increase above that minimum requirement and improve Virtual SAN performance.

Steps to Implement the Correct I/O Controller Driver for the Cisco 12G SAS Modular Raid Controller for Virtual SAN

This is my third post this week, possibly a record for me. All three are centered around ensuring the correct firmware and drivers are installed and running. The content of this post was created by my colleague, David Boone, who works with VMware customers to ensure successful Virtual SAN deployments. When it comes to VSAN, its important to use qualified hardware but equally important to make sure the correct firmware and drivers are installed.

Download the Correct I/O Controller Driver

Navigate to the VMware Compatibility Guide for Virtual SAN, scroll down and select “Build Your Own based on Certified Components”, then find the controller in the database. Here’s the link for the Cisco 12G SAS Modular Raid Controller and the link to download the correct driver for it (as of Nov. 20, 2015): https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESX55-LSI-SCSI-MEGARAID-SAS-660606001VMW&productId=353

Install the Correct Driver

Use your favorite way to install the driver. This might include creating a custom vSphere install image to deploy on multiple hosts, rolling out via vSphere Update Manager (VUM), or manually installing on each host.

Read more…

Verifying the Correct Version of the Cisco UCS C240 Server I/O Controller Firmware – Cisco 12G SAS Modular Raid Controller

Today I was working with Cisco to setup UCS C240 servers for Virtual SAN. As part of the process we needed to verify the Cisco 12G SAS Modular Raid Controller had the correct Firmware Version.

First we went to the VMware Compatibility Guide for Virtual SAN, navigated to the bottom of the page to the link for Build Your Own based on Certified Components. Under “Search For:” we selected “I/O Controller” and under “Brand Name:” we selected “Cisco” and found the listing for the Cisco 12G SAS Modular Raid Controller. It requires Firmware Version 4.270.00-4238.

vcg-Cisco 12G

Next we went into the Cisco UCS Manager and navigated to the Host Firmware Packages and found that the Storage Controller Firmware Package was 24.7.0-0047.

CiscoUCS1

Through UCS Manager there is no way to get the I/O Controller Firmware Version. So, we had to reboot the host and hit “CTRL-R” to get into the Cisco 12G SAS Modular Raid Controller Bios Configuration Utility.

CiscoUCS2

From here we hit CTRL-N to get into Properties.

CiscoUCS3

On this screen you can see:

Package: 24.7.0-0047

FW Version: 4.270.00-4238

Thus, we were able to confirm that we had the correct Firmware on the I/O Controller. If the FW Version was different than what VMware Virtual SAN supports, you would need to download the correct Firmware Package from Cisco and upgrade.

I hope this helps others save time trying to verify Firmware Versions. Thanks to my VMware Virtual SAN colleague, David Boone, who did most of the work that led to this post and our friends at Cisco for being a great partner and helping navigate UCS Manager and grabbing screenshots.

vSAN

V-SAN Logo 2 Final.jpgHere is a list of my VSAN related posts (most recent at the top)

Virtual SAN – Disk Hotplug

Yesterday the VMware Compatibility Guide for Virtual SAN was significantly updated to reflect recent certification completed by the team. I should point out that one of the great things about Virtual SAN is that it is a true Software Defined Storage solution in that VMware doesn’t force specific hardware upon customers. This being said, we know customers want and need quality and reliable storage solutions and that there are some low quality components out there that could cause undesirable results with Virtual SAN. For this reason, the team has taken on the daunting task of maintaining a Compatibility Guide for Virtual SAN made up of many hardware vendor IO Controllers, SSD, PCIe, and HDD’s. They do a great job managing the unlimited combinations and handling the requests from the field.

Some of the recent things added that I know many customers have been waiting for are:

Stay tuned for more updates in July.

One thing you might see as you browse various components (IO Controllers, SSD, PCIe, and HDD’s) on the VMware Compatibility Guide for Virtual SAN are:

  • Hot-plug
  • Hot-plug Drive
  • Hot-plug Hard Drive
  • Hotplug certification for this controller is not complete and is currently in progress.
  • Disk hotplug feature is not supported on the driver.
  • The driver does not support SSD hot unplug. Please refer to KB2092959 at http://kb.vmware.com/kb/2092959for more details.

Many customers have asked me what “hotplug” means or misinterpret that drives cannot be replaced without downtime. Here’s how I have responded when asked about this:

The VMware Compatibility Guide (VCG) for Virtual SAN lists the supported components (IO Controllers, SSD, and HDD) for use in Virtual SAN.  In the case of the IO Controllers it is a combination of the controller Firmware and ESXi Device Driver that are qualified for VSAN.  In the case of SSD, PCIe, or HDD’s its the firmware that is qualified.  VMware and our technology partners puts these through rigorous tests to make sure they are enterprise ready.

One of the many tests that are done is “Disk hotplug”.  In this test, VMware starts with a perfectly good vSphere cluster with VSAN enabled and I/O workloads running. Then they walk up to the hardware and yank a HDD or SSD out of one of the hosts.  It should be noted that this scenario would likely never happen on purpose in production.  Its possible to accidentally do it but if following proper maintenance procedures it should never happen. Especially now that vSphere 6.0 / VSAN 6.0 has the ability to blink the LED of a failed disk to properly identify it.  When this “failure” test is executed, some IO Controllers, SSD’s, and HDD’s do not behave properly.  The issue can be due to the firmware/driver combination provided to VMware by the IO Controller manufacturer or could be a problem with the HDD or SSD firmware. The VMware team works with the manufacturers to identify the issues and plan for resolution. Rather than not list the component on Compatibility Guide, VMware chooses to list it but caution against this unlikely event.

By the way, VSAN is smart enough to know the difference between a yanked drive and a drive that is degraded or failed, and triggers different events accordingly.  Since yanking a drive does not actually simulate a failed drive, it’s not really a valid failed drive test. Thus, the hotplug warnings do not apply to a more realistic scenario where a drive fails or has sector errors for instance.  In these scenarios, even the components that fail the hoplug test will behave properly and as expected.  Meaning, once VSAN detects that the disk is bad and marks it as a failed drive, the drives can be evacuated from VSAN using the vSphere Web Client, then technicians are free to remove it from a live system and replace it with a new one. This hardware replacement process can be different for every vendor.

I hope this helps clarify why we list the “hotplug” warnings on various components.

VSAN Sessions at VMworld 2014

I’m really looking forward to attending VMworld 2014 in a few weeks.  Its a great time to catch up with friends and meet new ones.  For me the week kicks off on Thursday at the VMware SE Tech Summit.  Then vOdgeball on Sunday afternoon with a team of former EMC vSpecialists.  This is a great event to help out a great cause, the Wounded Warrior Project.  We’ll also be honoring our two vSpecialist friends, Jim Ruddy and Stephen Hunt, that were recently involved in a tragic accident.  Then the main event starts Sunday night.  It should be fun and exhausting.

Virtual SAN (VSAN) is sure to be one of the highlights of the show.

Read more…

Configuring LSI MegaRAID 9271CV-81 I/O Controller for VSAN

A few colleagues of mine recently worked with a customer deploying VMware Virtual SAN (VSAN) with Cisco UCS hosts using the LSI MegaRAID 9271CV-81 I/O Controller and documented the configuration choices below. Note, in general the VSAN guidance has been to disable all controller cache so these choices follow this theme. Also note that we are not LSI experts and would welcome feedback from others on their experience with other settings.

Change the default settings in the screenshot below to the following:

  • Access = RW
  • I/O = Direct
  • Read = Disable
  • Disk Cache = Disabled
  • Disable BGI = No
  • Default Write = Write Through

LSI MegaRAID BIOS Config 01

* All settings can be changed on the fly or using storcli for VMware:

http://www.lsi.com/downloads/Public/RAID%2520Controllers/RAID%2520Controllers%2520Common%2520Files/1.09.13_StorCLI.zip 

* User guide

http://www.lsi.com/downloads/Public/RAID%20Controllers/RAID%20Controllers%20Common%20Files/51530-00_Rev_L.zip

Other versions of the MegaRAID controller might have a screen that looks something like the one below:

LSI MegaRAID BIOS Config 02

Thanks to my colleagues Justin Beck and Jason Burroughs for documenting and sharing their experience.