FlexPod and UCS – where are we now?

I have been working on FlexPod for about a year now, and on UCS for a couple of years; in my opinion it’s great tech and takes a lot of the pain away from designing new data center infrastructure solutions, taking away the guesswork, and bringing inherent simplicity, reliability and resilience to the space. Over the last year or so, there has been a shift in the Cisco Validated Designs (CVDs) coming out of Cisco, and as things have moved forward, there is a noticeable shift in the way things are going with FlexPod. I should note that some of what I discuss here is already clear, and some is mere conjecture. I think the direction things are going in with FlexPod is a symptom of changes in the industry, but it is clear that converged solutions are becoming more and more desirable as the Enterprise technology world moves forward.

So what are the key changes we are seeing?

The SDN elephant in the room 

Cisco’s ACI (Application-centric Infrastructure) has taken a while to get moving; the replacement of existing 3-tier network architecture with a leaf-spine network is something which is not going to happen overnight. The switched fabric is arguably a great solution for modern data centers, where east-west traffic is the bulk of network activity, and Spanning Tree Protocol continues to be the bane of network admins, but its implementation often requires either green-field deployments, or forklift replacement of large portions of the existing core networking of a data center.

That’s not to say ACI is not doing OK in terms of sales; Cisco’s figures, and case studies, seem to show that there is uptake, and some large customers taking this on. So how does this fit in with FlexPod? Well, the Cisco Validated Designs (CVDs) released over the last 12 months have all included the new Nexus 9000 series switches, rather than the previous stalwart Nexus 5000 series. These are ACI based switches, which are also able to operate in the ‘legacy’ NX-OS mode. Capability wise, in the context of FlexPod, there is not a lot of difference, they now have FCoE support, can do vPC, QoS, Layer 3, and all the good stuff we come to expect from Nexus switches.

So from what I can gather, the inclusion of 9K switches in the FlexPod line (outside of the FlexPod with ACI designs), is there to enable FlexPod customers to more easily move into the leaf/spine ACI network architecture at a later date, should they wish to do this. This makes sense, and the pricing on the 9Ks being used looks favourable over the 5Ks, so this is a win-win for customers, even if they don’t eventually decide to go with ACI.

40GbE standard 

Recent announcements around the Gen 3 UCS Fabric Interconnects have revealed that 40GbE is now going to be the standard for UCS connectivity solutions, and the new chassis designs show 4 x 40GbE QSFP connections, totaling 320Gbps total bandwidth per chassis, this is an incredible throughput, and although I can’t see 99% of customers going anywhere near these levels, it does help to strengthen the UCS platform’s use cases for even the most high performance environments, and reduces the requirement for Infiniband type solutions for high throughput environments.

Another interesting point, and following on from the ACI ramblings above, is that the new 6300 series Fabric Interconnects are now based on the Nexus 9300 switching line, rather than the Nexus 5K based 6200 series. This positions them perfectly to act as a leaf in an ACI fabric one day, should this become the eventual outcome of Cisco’s strategy.

HTML5 FTW! 

With the announcements about the new UCS generation, came the news that from UCS firmware version 3.1, the software for UCS would now be unified for UCS classic, UCS Mini, and the newish M-series systems, this simplifies things for people looking at version numbers and how they relate to the relevance of the release, and means that there should now be relative feature parity across all footprints of UCS systems.

The most exciting, if you have experienced long running pain with Java, is that the new release incorporates the HTML5 interface, which has been seen on UCS Mini since its release. I’m sure this will bring new challenges with it, but for now at least, something fresh to look forward to for those running UCS classic.

FlexPod Mini – now not so mini 

 

FlexPod Mini is based on the UCS Mini release, which came out around 18 months ago, this is based on the I/O Modules (or FEXs) in the UCS 5108 chassis, being replaced with UCS 6324 pocket sized Fabric Interconnects, ultimately cramming a single chassis of UCS, and the attached network equipment into just 6 rack units. This could be expanded with C-series servers, but the scale for UCS blades, was strictly limited to the 8 blade limit of a standard chassis. With the announcement of the new Fabric Interconnect models came the news of a new ‘QSFP Scalability Port License’, which allows the 40GbE port on the UCS 6324 FI to be utilized with a 1 x QSFP to 4 x SFP+ cable, to add another chassis to the UCS Mini.

Personally I haven’t installed a UCS Mini, but the form factor is a great fit for certain use cases, and the more flexible this is, the more desire there will be to use this solution. For FlexPod, this ultimately means more suitable use cases, particularly in the ROBO (remote office/branch office) type scenario.

What’s Next? 

So with the FlexPod now having new switches, and new UCS hardware, it seems something new from NetApp is next on the list for FlexPod. The FAS8000 series is a couple of years old now, so we will likely see a refresh on this at some point, probably with 40GbE on board, more flash options, and faster CPUs. The recent purchase of SolidFire by NetApp will also quite probably see some kind of SolidFire based FlexPod CVD coming out of the Cisco/NetApp partnership in the near future.

We are also seeing the release of some exciting (or at least as exciting as these things can be!) new software releases this year: Windows Server 2016, vSphere 6.5 (assuming this will be revealed at VMworld), and OpenStack Mitaka, all of which will likely bring new CVDs.

Automating UCS System Builds

A project I have been working on recently is the full automation of the Cisco UCS system build. Automation comes from a desire to not sit clicking buttons; that’s a great way to learn how a system fits together, but once you have done it once or twice, it is no longer exciting. A look at the FlexPod Cisco Validated Design (CVD) shows around 50 pages of configuration steps. This is largely the same, regardless of the client, and will take at least 3 hours of clicking buttons. 

This shows this to be a prime candidate for automation. There are a few options available here for the automation of this task:

  • Cisco PowerTool for UCS
  • Python API for Cisco UCS
  • Altering of XML backups and restoration onto the UCS system

Altering an exported XML configuration backup, to fit the customer’s solution works, but it is not particularly neat or nice, and will result in replacing all of the system specific information through copy/paste, or a more neat solution parsing the XML and replacing elements. This is not something I really want to do, and it does not leave us with a particularly customisable solution.

The Python API is extensive, and has a tonne of documentation around it. I have run through the Codecademy course for Python, and understand the basics, but I come from a Windows administrative background, and I am not comfortable enough that i want to do this from scratch for UCS. This is something to put down for the future, as my Python knowledge grows. The great advantage in using Python is that it is platform agnostic, so I could run this from a Mac, Linux, or Windows environment (as long as i have the Python packages installed). Sounds great, but the documentation from Cisco around Managed Objects melts my brain, so this is something i discounted for now.

Luckily, Cisco have done a fantastic job with their PowerShell library for Cisco UCS. This is simple to get started with, and the great thing about scripting with PowerShell is that anyone can read it and figure out what the script is doing. As an infrastructure engineer, with a background in PowerShell and VMware’s equally excellent PowerCLI PowerShell module. this was the natural fit for me.

So where did I, and should you (in my opinion), start with automating your Cisco UCS builds?

The first element of this i have already mentioned, get Cisco PowerTool. This is available for download from Cisco’s site, and once you have the MSI installed, you can launch your PowerShell console from Windows, and run ‘Import-Module CiscoUCSPS’ to import the module. You now have the power at your fingertips.

The next good place to start, whether you know a lot about UCS or not, is to get the UCS Platform Emulator, and get this deployed. This comes as an OVA appliance, which you can deploy to VMware Player, VMware Workstation, ESXi, or any other compatible Type 1 or Type 2 hypervisor. Once this has booted, go to the console and give it an IP (if it didn’t already get one from your DHCP server).

It should be noted that there are currently 3 active versions of the UCSPE available: one for UCS standard, one for UCS M-series servers, and one for UCS Mini. Make sure you get the right one for what you are wanting to automate. In this example we are looking at standard UCS, so grab that one, and spin it up.

Now we have the PE up and IP’d, so open your web browser and go to the IP. Here you can amend the hardware configuration to match your environment if that is useful. Most of the automation you will be doing is creating policies, configuring the Fabric Interconnects, creating service profiles, so the exact hardware configuration is not hugely important, and customising the hardware in here can be quite time consuming and frustrating as the interface is clunky and unintuitive.

So now we have our Platform Emulator stood up, we can connect in to it using PowerTool. Open a PowerTool window and enter ‘Connect-Ucs <IP>’, you will be prompted for credentials. Default on the Platform Emulator is admin/admin, so enter this and you are good to go.

There are hundreds of cmdlets in the UCS PowerTool module, I am not going to go through them here, but I will show a couple of tactics for, firstly finding what you need in the set of cmdlets, and secondly, for letting PowerTool do the heavy lifting, and produce the script for the build for you.

So lets start with looking for commands in the PowerTool module, we can run ‘Get-Command -Module CiscoUcsPS’, which will give us a list of all the commands in the module. This is a great place to start, and in general the cmdlets are named fairly sensibly, although due to the way PowerTool was converted from the Python API model, some of these are pretty long.

Screen Shot 2015-11-07 at 08.22.46

If we want to search for something more specific, we can use ‘Get-Command -Module CiscoUcsPS *blade*’, for example, to search for all commands in this module with the word ‘blade’ in them. This narrows the search to something you might actually want to do at that time.

Once you have located the cmdlet you want, in this case we will use ‘Get-UcsBlade’ which lists all the blade servers installed in your UCS environment, you can get more information about that by running ‘Get-Help Get-UcsBlade’ followed by one of three optional parameters: ‘-full’, ‘-detailed’ or ‘-examples’. If you run this with no parameters, you will get the basic description and the syntax of the command. If you go for ‘-examples’ you will get examples of the command usage. Entering ‘-detailed’ will go into details on each parameter, what data type they are, whether they are mandatory or optional, basically a tonne of information you probably won’t need (although it’s useful to know this is there when you need it), and ‘-full’ will show you the combination of all three of these.

One thing to be aware of, is that different vendors have different qualities of PowerShell modules. I am a fairly heavy user of the Cisco, VMware, NetApp, and the core Microsoft cmdlets. The VMware ones tend to be very well documented, well presented, and have a tonne of options, while not always providing specific commands which you may need (although using ‘Get-View’ we can tap into anywhere in the API, a story for another day I think). The NetApp commands have excellent examples, but general documentation and community does not really happen, and there are quite a few absent commands which mean you can’t always do what you need to do. The Cisco PowerTool pack has a huge breadth of available commands, which means you can do pretty much anything in there, but they don’t always have examples in the help for the cmdlets, and some of the detailed help is lacking description so leaves success with some commands subject to your trial and error.

Screen Shot 2015-11-07 at 08.23.37

So now you can find the commands you need, and find out how to use them. Well because PowerShell is a declarative tool, we can run single commands from the shell, so if you run ‘Get-UcsBlade’ while connected to your PE, you will see the list of blade servers attached to the system. This list will likely be a few pages so you can trim it down to a table by entering ‘Get-UcsBlade | Format-Table’, or ‘Get-UcsBlade | ft’ for short, which will present the same output with only a few select fields in a nice small table. There are a load of ways of playing with this output to customise it for what you need. i’m not going to go into that here, but suffice to say this is a good way to get information out of your UCS system.

By using the commands, you can build your UCS system piece by piece, this is going to take you a while as you get used to the nuances of PowerTool, and if you have the patience for this then great, you will be far stronger at the end of it, but when I started I used a different method which I will now describe.

One of the great tools we have in PowerTool, is a cmdlet called ‘ConvertTo-UcsCmdlet’. This is a life saver, as it lets you automate, without really needing to know how to write PowerShell scripts. So it works like this.

Once you are connected to your PE through PowerTool (using Connect-Ucs <IP>), you enter the ‘ConvertTo-UcsCmdlet’ command, and the command prompt will disappear. If you need to get out of this mode, just press Ctrl+C, but just leave it for now.

Open up your UCS Manager through your PE’s web front end, and log in. Now go and create something, say a Network Control Policy, for something simple, click OK to save your new policy, and go back to your PowerShell window.

You should see what our magical cmdlet has done, it should have dumped the PowerTool commands for what you just did in the window. Now you can just copy and paste this cmdlet into your favourite text editor, and voila, you don’t need to click buttons to do that again, you can just use this piece of script.

Through doing this, you can build up the full configuration in a text file, name the file with a .ps1 extension, and when you are ready to test it you can factory reset the PE through the web interface and run it again. In a few hours you can quickly create the full build from start to finish using PowerTool.

There are some things which ConvertTo-UcsCmdlet will not convert, creation of Service Profiles is one, but if you look around online there are plenty of good people sharing scripts which can be modified for your purposes.

Hope this helps people, it certainly changed things for me, taking a 3 hour build down to around 30 seconds. Once you have this up and running, you can take what you have and replace the specific elements, say sub-organisation name, and replace them with variables, this script can then be reused again and again. This is a quick way to open your eyes to the power of scripting.