/ Cisco

Cisco UCS B200 M3 and FlexFlash

So if you found this post you must be having a problem with your Cisco UCS B200 M3 server with FlexFlash. It doesn't matter the OS you are trying to install on the SD cards, but most of the installs in this case will be VMware ESXi. If you are like me, I love automation and my environment at the place that I work act, I have designed my Cisco UCS and VMware system to be "plug and play" so that all someone has to do is plug a server into the Cisco UCS domain, it gets pre-configured with a server profile on Cisco UCS, and then automatically setup/install VMware onto the SD cards so that when the next time it boots it boots off the SD cards as I do not like the fiber channel/iSCSI boot settings. Most installs in the Cisco UCS Blade environment recommended and you will see a lot of Fiber channel setups on your storage array.

A little background, I find a problem with this setup even with two interconnects and many paths to your storage array, you now have to take the storage array as a point of failure, and then you also lose critical storage space in your storage and most people will provision that 60GB LUN when you just need less than 16GB for the install. However no matter the size of any LUN, you have to span many disks just to have a failure and that is a waste and storing more than one LUN one a volume could even be worse as many servers could just die if the volume fails, and then you have to replicate that volume to another storage array on top of snapshots. Just tons of overhead.

So here comes the power of SD cards and the problems you can have. One of the main issues is that the B200 M3 server cannot support SD cards over 32GB. Cisco only sells 64, 32, and 16GB cards so you have limited options on size. 32GB gives you more "dumps" and might be useful if you have lots of automating to your ESXi image plus the core dump of VMware when VMware crashes. So if you have 4 Blade servers, you need 8 SD cards for the FlexFlash to work as you always should set this in a RAID 1 format.

One of the main issues of the Cisco B200 M3 servers is the FlexFlash controller version. If you are up to the latest version of the firmware, and you follow the offical FlexFlash guide it seems simple enough. However the is a problem if your FlexFlash controller is at version 1.2 build 258-2 a problem exists. While the official guide will make your SD cards show up as "Connected Partition Over USB To Host" you will have a problem when trying to install VMware either with Autodeploy and/or regular ISO installation. This is because the controller doesn't fully format/partision the SD cards with a service profile that set to "Flex Flash Scrub" set to True. So here is the simple, yet not so obvious fix.

Cisco UCS FlexFlash Controller Information

After finding this post I set my "default" (builtin) policy to FlexFlash Scrub to True.

Default Scrub Policy

Then I set the M3 server with no associating to a server profile and did a 2x server re-Acknowledgement on the server (of course waiting at least two full cycles to compelte.). After that, I set the server back to the service profile that has my VMware ESXi AutoDeploy settings (in this case your server profile). My server booted with PXE, booted up with the VMware Auto Deploy, and as part of the Autodeply host profile it had copied/installed VMware ESXi on to the SD cards without issue.

Mmost people will think that setting the service profile to scrub to "true" does the trick. This is a false glimpse of reaility that the SD cards are working, but in reality, they are not.

This was a hard error to find as the documentation doesn't really make this clear plus it seems the coders of the FlexFlash put this hard-coded setting that the 'default' FlexFlash has to be used which is never a good idea. As of April 13th, 2018, I do have a open case still open with this issue, but I setup some dialog so that the developers could also look into it.

If you happen to have Cisco M4's as well up to the latest version, you will also know that the FlexFlash when it's empty does not get past the SD card check if it's empty. You have to press a key for it to continue. Another failure on the Cisco UCS development team.

In my next post, I will show the "Does" and "Don'ts" of how to set up the best VMware AutoDeply in the Cisco UCS cluster.

Till next time...

Shane

Shane

Shane is the founder and author of 'the rabbit hole'. By day, Shane is a System and Network Engineer living and working in the NYC area. Free time includes movies, escaping to the non-tech world.

Read More