Deploying NX-OSv 9000 on vSphere

Cisco have recently released (1st March 2017) an updated virtual version of their Nexus 9K switch, and the good news is that this is now available as an OVA for deployment onto ESXi. We used to use VIRL in a lab, which was fine until a buggy earlier version of the virtual 9K was introduced which prevented core functionality like port channels. This new release doesn’t require the complex environment that VIRL brings, and lets you deploy a quick pair of appliances in vPC to test code against.

The download is available here, and while there are some instructions available, I did not find them particularly useful in deploying the switch to my ESXi environment. As a result, I decided to write up how I did this to hopefully save people spending time smashing their face off it.

Getting the OVA

NOTE: you will need a Cisco login to download the OVA file. My login has access to a bunch of bits so not sure exactly what the requirements are around this.

There are a few versions available from the above link, including a qcow2 (KVM) image, a .vmdk file (for rolling your own VM), a VirtualBox image (for use with VirtualBox and/or Vagrant), and an OVA (for use with Fusion, Workstation, ESXi).

Once downloaded we are ready to deploy the appliance. There are a few things to bear in mind here:

  1. This can be used to pass VM traffic between virtual machines: there are 6 connected vNICs on deployment, 1 of these simulates the mgmt0 port on the 9K, and the other 5 are able to pass VM traffic.
  2. vNICs 2-6 should not be attached to the management network (best practice)
  3. We will need to initially connect over a virtual serial port through the host, this will require opening up the ESXi host firewall temporarily

Deploying the OVA

You can deploy the OVA through the vSphere Web Client, or the new vSphere HTML5 Web Client, I’ve detailed how to do this via PowerShell here, because who’s got time for clicking buttons?


# Simulator is available at:
# Filename: nxosv-final.7.0.3.I5.2.ova
# Documentation:

Function New-SerialPort {
  # stolen from
  ) #end
  $dev = New-Object VMware.Vim.VirtualDeviceConfigSpec
  $dev.operation = "add"
  $dev.device = New-Object VMware.Vim.VirtualSerialPort
  $dev.device.key = -1
  $dev.device.backing = New-Object VMware.Vim.VirtualSerialPortURIBackingInfo
  $dev.device.backing.direction = "server"
  $dev.device.backing.serviceURI = "telnet://"+$hostIP+":"+$prt
  $dev.device.connectable = New-Object VMware.Vim.VirtualDeviceConnectInfo
  $dev.device.connectable.connected = $true
  $dev.device.connectable.StartConnected = $true
  $dev.device.yieldOnPoll = $true

  $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
  $spec.DeviceChange += $dev

  $vm = Get-VM -Name $vmName

# Variables - edit these...
$ovf_location = '.\nxosv-final.7.0.3.I5.1.ova'
$n9k_name = 'NXOSV-N9K-001'
$target_datastore = 'VBR_MGTESX01_Local_SSD_01'
$target_portgroup = 'vSS_Mgmt_Network'
$target_cluster = 'VBR_Mgmt_Cluster'

$vi_server = ''
$vi_user = 'administrator@vsphere.local'
$vi_pass = 'VMware1!'

# set this to $true to remove non-management network interfaces, $false to leave them where they are
$remove_additional_interfaces = $true

# Don't edit below here
Import-Module VMware.PowerCLI

Connect-VIServer $vi_server -user $vi_user -pass $vi_pass

$vmhost = $((Get-Cluster $target_cluster | Get-VMHost)[0])

$ovfconfig = Get-OvfConfiguration $ovf_location

$ovfconfig.NetworkMapping.mgmt0.Value = $target_portgroup
$ovfconfig.NetworkMapping.Ethernet1_1.Value = $target_portgroup
$ovfconfig.NetworkMapping.Ethernet1_2.Value = $target_portgroup
$ovfconfig.NetworkMapping.Ethernet1_3.Value = $target_portgroup
$ovfconfig.NetworkMapping.Ethernet1_4.Value = $target_portgroup
$ovfconfig.NetworkMapping.Ethernet1_5.Value = $target_portgroup
$ovfconfig.DeploymentOption.Value = 'default'

Import-VApp $ovf_location -OvfConfiguration $ovfconfig -VMHost $vmhost -Datastore $target_datastore -DiskStorageFormat Thin -Name $n9k_name

if ($remove_additional_interfaces) {
  Get-VM $n9k_name | Get-NetworkAdapter | ?{$_.Name -ne 'Network adapter 1'} | Remove-NetworkAdapter -Confirm:$false

New-SerialPort -vmName $n9k_name -hostIP $($vmhost | Get-VMHostNetworkAdapter -Name vmk0 | Select -ExpandProperty IP) -prt 2000

$vmhost | Get-VMHostFirewallException -Name 'VM serial port connected over network' | Set-VMHostFirewallException -Enabled $true

Get-VM $n9k_name | Start-VM

This should start the VM, we will be able to telnet into the host on port 2000 to reach the VM console, but it will not be ready for us to do that until this screen is reached:


Now when we connect we should see:


At this point we can enter ‘n’ and go through the normal Nexus 9K setup wizard. Once the management IP and SSH are configured you should be able to connect via SSH, the virtual serial port can then be removed via the vSphere Client, and the ‘VM serial port connected over network’ rule should be disabled on the host firewall.

Pimping things up

Add more NICs

Obviously here we have removed the additional NICs from the VM, which makes it only talk over the single management port. We can add a bunch more NICs and the virtual switch will let us use them to talk on. This could be an interesting use case to pass actual VM traffic through the 9K.

Set up vPC

The switch is fully vPC (Virtual Port Channel) capable, so we can spin up another virtual N9K and put them in vPC mode, this is useful to experiment with that feature.

Bring the API!

The switch is NXAPI capable, which was the main reason for me wanting to deploy it, so that I could test REST calls against it. Enable NXAPI by entering the ‘feature nxapi’ commmand.


Hopefully this post will help people struggling to deploy this OVA, or wanting to test out NXOS in a lab environment. I found the Cisco documentation a little confusing so though I would share my experiences.

FlexPod and UCS – where are we now?

I have been working on FlexPod for about a year now, and on UCS for a couple of years; in my opinion it’s great tech and takes a lot of the pain away from designing new data center infrastructure solutions, taking away the guesswork, and bringing inherent simplicity, reliability and resilience to the space. Over the last year or so, there has been a shift in the Cisco Validated Designs (CVDs) coming out of Cisco, and as things have moved forward, there is a noticeable shift in the way things are going with FlexPod. I should note that some of what I discuss here is already clear, and some is mere conjecture. I think the direction things are going in with FlexPod is a symptom of changes in the industry, but it is clear that converged solutions are becoming more and more desirable as the Enterprise technology world moves forward.

So what are the key changes we are seeing?

The SDN elephant in the room 

Cisco’s ACI (Application-centric Infrastructure) has taken a while to get moving; the replacement of existing 3-tier network architecture with a leaf-spine network is something which is not going to happen overnight. The switched fabric is arguably a great solution for modern data centers, where east-west traffic is the bulk of network activity, and Spanning Tree Protocol continues to be the bane of network admins, but its implementation often requires either green-field deployments, or forklift replacement of large portions of the existing core networking of a data center.

That’s not to say ACI is not doing OK in terms of sales; Cisco’s figures, and case studies, seem to show that there is uptake, and some large customers taking this on. So how does this fit in with FlexPod? Well, the Cisco Validated Designs (CVDs) released over the last 12 months have all included the new Nexus 9000 series switches, rather than the previous stalwart Nexus 5000 series. These are ACI based switches, which are also able to operate in the ‘legacy’ NX-OS mode. Capability wise, in the context of FlexPod, there is not a lot of difference, they now have FCoE support, can do vPC, QoS, Layer 3, and all the good stuff we come to expect from Nexus switches.

So from what I can gather, the inclusion of 9K switches in the FlexPod line (outside of the FlexPod with ACI designs), is there to enable FlexPod customers to more easily move into the leaf/spine ACI network architecture at a later date, should they wish to do this. This makes sense, and the pricing on the 9Ks being used looks favourable over the 5Ks, so this is a win-win for customers, even if they don’t eventually decide to go with ACI.

40GbE standard 

Recent announcements around the Gen 3 UCS Fabric Interconnects have revealed that 40GbE is now going to be the standard for UCS connectivity solutions, and the new chassis designs show 4 x 40GbE QSFP connections, totaling 320Gbps total bandwidth per chassis, this is an incredible throughput, and although I can’t see 99% of customers going anywhere near these levels, it does help to strengthen the UCS platform’s use cases for even the most high performance environments, and reduces the requirement for Infiniband type solutions for high throughput environments.

Another interesting point, and following on from the ACI ramblings above, is that the new 6300 series Fabric Interconnects are now based on the Nexus 9300 switching line, rather than the Nexus 5K based 6200 series. This positions them perfectly to act as a leaf in an ACI fabric one day, should this become the eventual outcome of Cisco’s strategy.


With the announcements about the new UCS generation, came the news that from UCS firmware version 3.1, the software for UCS would now be unified for UCS classic, UCS Mini, and the newish M-series systems, this simplifies things for people looking at version numbers and how they relate to the relevance of the release, and means that there should now be relative feature parity across all footprints of UCS systems.

The most exciting, if you have experienced long running pain with Java, is that the new release incorporates the HTML5 interface, which has been seen on UCS Mini since its release. I’m sure this will bring new challenges with it, but for now at least, something fresh to look forward to for those running UCS classic.

FlexPod Mini – now not so mini 


FlexPod Mini is based on the UCS Mini release, which came out around 18 months ago, this is based on the I/O Modules (or FEXs) in the UCS 5108 chassis, being replaced with UCS 6324 pocket sized Fabric Interconnects, ultimately cramming a single chassis of UCS, and the attached network equipment into just 6 rack units. This could be expanded with C-series servers, but the scale for UCS blades, was strictly limited to the 8 blade limit of a standard chassis. With the announcement of the new Fabric Interconnect models came the news of a new ‘QSFP Scalability Port License’, which allows the 40GbE port on the UCS 6324 FI to be utilized with a 1 x QSFP to 4 x SFP+ cable, to add another chassis to the UCS Mini.

Personally I haven’t installed a UCS Mini, but the form factor is a great fit for certain use cases, and the more flexible this is, the more desire there will be to use this solution. For FlexPod, this ultimately means more suitable use cases, particularly in the ROBO (remote office/branch office) type scenario.

What’s Next? 

So with the FlexPod now having new switches, and new UCS hardware, it seems something new from NetApp is next on the list for FlexPod. The FAS8000 series is a couple of years old now, so we will likely see a refresh on this at some point, probably with 40GbE on board, more flash options, and faster CPUs. The recent purchase of SolidFire by NetApp will also quite probably see some kind of SolidFire based FlexPod CVD coming out of the Cisco/NetApp partnership in the near future.

We are also seeing the release of some exciting (or at least as exciting as these things can be!) new software releases this year: Windows Server 2016, vSphere 6.5 (assuming this will be revealed at VMworld), and OpenStack Mitaka, all of which will likely bring new CVDs.

In the zone…basic zoning on Cisco Nexus switches

In this post I look to go over some basic FCoE zoning concepts for Cisco Nexus switches, although FCoE has not really captured the imagination of the industry, it is used in a large number of Cisco infrastructure deployments, particularly around Nexus and UCS technologies. My experience is mostly based on FlexPods where we have this kind of design (this shows FCoE connectivity only, to keep things simple):

Screen Shot 2015-11-08 at 18.38.23

Zoning in a FlexPod is simple enough, we may have a largish number of hosts, but we are only zoning on two switches, only have 4 or 8 FCoE targets, depending on our configuration. In fact the zoning configuration can be fairly simply automated using PowerShell by tapping into the NetApp, Cisco UCS, and NX-OS APIs. The purpose of what we are doing here though is to describe the configuration steps required to complete the zoning.

The purpose of zoning is to restrict access to a LUN (Logical Unit Number), or essentially a Fibre Channel block device, on our storage, to one or more access devices, or hosts. This is useful in the case of boot disks, where we only ever want a single host accessing that device, and in the case of shared data devices, like cluster shared disks in Microsoft Clustering, or VMFS datastore in the VMware world, where we only want a subset of hosts to be able to access the device.

I found configuring zoning on a Cisco switch took a bit of getting my head around, so hopefully the explanation below will help to make this simpler for someone else.

From the Cisco UCS (or any server infrastructure you are running), you will need to gather a list of the WWPNs for the initiators wanting to connect to the storage. These will be in the format of 50:02:77:a4:10:0c:4e:21, this being a 16 byte hexadecimal number. Likewise, you will need to gather the WWPNs from your storage (in the case of FlexPod, your NetApp storage system).

Once we have these, we are ready to do our zoning on the switch. When doing the zoning configuration there are three main elements of configuration we need to understand:

  1. Aliases – these match the WWN to a friendly name, and sit in the device alias database on the switch. You can get away without using this, and just use the native WWNs later, but this will make things far more difficult should something go wrong. So basically these just match WWNs to devices.
  2. Zones – these logically group initiators and targets together, meaning that only the device aliases listed in the zone are able to talk to one another. This provides security and ease of management, a device can exist in more than one zone.
  3. Zonesets – this groups together zones, allowing the administrator to bring all the zones online or offline together. Only one zoneset can be active at a time.

On top of this, there is one more thing to understand when creating zoning on our Nexus switch, and that is the concept of a VSAN. A VSAN, or Virtual Storage Area Network, is the Fibre Channel equivalent of a VLAN. It is a logical collection of ports which together form a single discrete fabric.

So let’s create a fictional scenario, and create the zoning configuration for this. We have a FlexPod with two Nexus 5k switches, with separate fabrics, as shown in the diagram above, meaning that our 0a ports on the servers only go to Nexus on fabric A, and 0b ports only go to Nexus on fabric B. Both of our e0c ports on our NetApp storage go to NexusA, and both our e0d ports go to NexusB:

AFAS01 – e0c, e0d

NAFAS02 – e0c, e0d

And 3 Cisco UCS service profiles, each with two vHBAs, wanting to access storage on these targets, these are created as follows:

UCSServ01 – 0a, 0b

UCSServ02 – 0a, 0b

UCSServ03 – 0a, 0b

So on NexusA, we need the following aliases in our database:

Device Port WWPN Alias Name
NAFAS01 e0c 35:20:01:0c:11:22:33:44 NAFAS01_e0c
NAFAS02 e0c 35:20:02:0c:11:22:33:44 NAFAS01_e0c
UCSServ01 0a 50:02:77:a4:10:0c:0a:01 UCSServ01_0a
UCSServ02 0a 50:02:77:a4:10:0c:0a:02 UCSServ02_0a
UCSServ03 0a 50:02:77:a4:10:0c:0a:03 UCSServ03_0a

And on NexusB, we need the following:

Device Port WWPN Alias Name
NAFAS01 e0d 35:20:01:0d:11:22:33:44 NAFAS01_e0d
NAFAS02 e0d 35:20:02:0d:11:22:33:44 NAFAS02_e0d
UCSServ01 0b 50:02:77:a4:10:0c:0b:01 UCSServ01_0b
UCSServ02 0b 50:02:77:a4:10:0c:0b:02 UCSServ02_0b
UCSServ03 0b 50:02:77:a4:10:0c:0b:03 UCSServ03_0b

And the zones we need on each switch are, firstly for NexusA:

Zone Name Members
UCSServ01_a NAFAS01_e0c



UCSServ02_a NAFAS01_e0c



UCSServ03_a NAFAS01_e0c



And for Nexus B:

Zone Name Members
UCSServ01_b NAFAS01_e0d



UCSServ02_b NAFAS01_e0d



UCSServ03_b NAFAS01_e0d



This gives us a zone for each server to boot from, allowing that vHBA on the server to boot from either of the NetApp interfaces that it will be able to see on its fabric. The boot order itself will be controlled from within UCS, by creating zoning for the server to boot on either fabric we create resilience. All of this is just to demonstrate how we construct the zoning configuration so things will no doubt be different in a different environment.

So now we know what we should have in our populated alias database, and our zone configuration, we just need to create our zoneset. Well, we will have one zoneset per fabric, so one for NexusA:

Zoneset Name Members
UCSZonesetA UCSServ01_a



And the zoneset for NexusB:

Zoneset Name Members
UCSZonesetB UCSServ01_b



Now we are ready to put this into some NXOS CLI, and enter this on our switches. The general commands for creating new aliases are:
device-alias database
device-alias name <alias_name> pwwn <device_wwpn>
device-alias commit

So for our NexusA, we do the following:
device-alias database
device-alias name NAFAS01_e0c pwwn 35:20:01:0c:11:22:33:44
device-alias name NAFAS02_e0c pwwn 35:20:02:0c:11:22:33:44
device-alias name UCSServ01_0a pwwn 50:02:77:a4:10:0c:0a:01
device-alias name UCSServ02_0a pwwn 50:02:77:a4:10:0c:0a:02
device-alias name UCSServ03_0a pwwn 50:02:77:a4:10:0c:0a:03
device-alias commit

And for Nexus B, we do:
device-alias database
device-alias name NAFAS01_e0d pwwn 35:20:01:0d:11:22:33:44
device-alias name NAFAS02_e0d pwwn 35:20:02:0d:11:22:33:44
device-alias name UCSServ01_0b pwwn 50:02:77:a4:10:0c:0b:01
device-alias name UCSServ02_0b pwwn 50:02:77:a4:10:0c:0b:02
device-alias name UCSServ03_0b pwwn 50:02:77:a4:10:0c:0b:03
device-alias commit

So that’s our alias database taken care of, now we can create our zones. The command set for creating a zone is:
zone name <zone_name> vsan <vsan_id>
member device-alias <device_1_alias>
member device-alias <device_2_alias>
member device-alias <device_3_alias>

I will use VSAN IDs 101 for fabric A, and 102 for fabric B. So here we will create our zones for NexusA:
zone name UCSServ01_a vsan 101
member device-alias NAFAS01_e0c
member device-alias NAFAS02_e0c
member device-alias UCSServ01_0a
zone name UCSServ02_a vsan 101
member device-alias NAFAS01_e0c
member device-alias NAFAS02_e0c
member device-alias UCSServ02_0a
zone name UCSServ03_a vsan 101
member device-alias NAFAS01_e0c
member device-alias NAFAS02_e0c
member device-alias UCSServ03_0a

And for NexusB:
zone name UCSServ01_b vsan 102
member device-alias NAFAS01_e0d
member device-alias NAFAS02_e0d
member device-alias UCSServ01_0b
zone name UCSServ02_b vsan 102
member device-alias NAFAS01_e0d
member device-alias NAFAS02_e0d
member device-alias UCSServ02_0b
zone name UCSServ03_b vsan 102
member device-alias NAFAS01_e0d
member device-alias NAFAS02_e0d
member device-alias UCSServ03_0b

So this is all of our zones created, now we just need to create and activate our zoneset and we have our completed zoning configuration. The commands to create and activate a zoneset are:
zoneset name <zoneset_name> vsan <vsan_id>
member <zone_1_name>
member <zone_2_name>
zoneset activate name <zoneset_name> vsan <vsan_id>

So now we have our NexusA configuration:
zoneset name UCSZonesetA vsan 101
member UCSServ01_a
member UCSServ02_a
member UCSServ03_a
zoneset activate name UCSZonesetA vsan 101

And our NexusB configuration:
zoneset name UCSZonesetB vsan 102
member UCSServ01_b
member UCSServ02_b
member UCSServ03_b
zoneset activate name UCSZonesetB vsan 102

So that’s how we compose our zoning configuration, and apply it to our Nexus switch. Hopefully this will be a useful reference on how to do this.

Automating UCS System Builds

A project I have been working on recently is the full automation of the Cisco UCS system build. Automation comes from a desire to not sit clicking buttons; that’s a great way to learn how a system fits together, but once you have done it once or twice, it is no longer exciting. A look at the FlexPod Cisco Validated Design (CVD) shows around 50 pages of configuration steps. This is largely the same, regardless of the client, and will take at least 3 hours of clicking buttons. 

This shows this to be a prime candidate for automation. There are a few options available here for the automation of this task:

  • Cisco PowerTool for UCS
  • Python API for Cisco UCS
  • Altering of XML backups and restoration onto the UCS system

Altering an exported XML configuration backup, to fit the customer’s solution works, but it is not particularly neat or nice, and will result in replacing all of the system specific information through copy/paste, or a more neat solution parsing the XML and replacing elements. This is not something I really want to do, and it does not leave us with a particularly customisable solution.

The Python API is extensive, and has a tonne of documentation around it. I have run through the Codecademy course for Python, and understand the basics, but I come from a Windows administrative background, and I am not comfortable enough that i want to do this from scratch for UCS. This is something to put down for the future, as my Python knowledge grows. The great advantage in using Python is that it is platform agnostic, so I could run this from a Mac, Linux, or Windows environment (as long as i have the Python packages installed). Sounds great, but the documentation from Cisco around Managed Objects melts my brain, so this is something i discounted for now.

Luckily, Cisco have done a fantastic job with their PowerShell library for Cisco UCS. This is simple to get started with, and the great thing about scripting with PowerShell is that anyone can read it and figure out what the script is doing. As an infrastructure engineer, with a background in PowerShell and VMware’s equally excellent PowerCLI PowerShell module. this was the natural fit for me.

So where did I, and should you (in my opinion), start with automating your Cisco UCS builds?

The first element of this i have already mentioned, get Cisco PowerTool. This is available for download from Cisco’s site, and once you have the MSI installed, you can launch your PowerShell console from Windows, and run ‘Import-Module CiscoUCSPS’ to import the module. You now have the power at your fingertips.

The next good place to start, whether you know a lot about UCS or not, is to get the UCS Platform Emulator, and get this deployed. This comes as an OVA appliance, which you can deploy to VMware Player, VMware Workstation, ESXi, or any other compatible Type 1 or Type 2 hypervisor. Once this has booted, go to the console and give it an IP (if it didn’t already get one from your DHCP server).

It should be noted that there are currently 3 active versions of the UCSPE available: one for UCS standard, one for UCS M-series servers, and one for UCS Mini. Make sure you get the right one for what you are wanting to automate. In this example we are looking at standard UCS, so grab that one, and spin it up.

Now we have the PE up and IP’d, so open your web browser and go to the IP. Here you can amend the hardware configuration to match your environment if that is useful. Most of the automation you will be doing is creating policies, configuring the Fabric Interconnects, creating service profiles, so the exact hardware configuration is not hugely important, and customising the hardware in here can be quite time consuming and frustrating as the interface is clunky and unintuitive.

So now we have our Platform Emulator stood up, we can connect in to it using PowerTool. Open a PowerTool window and enter ‘Connect-Ucs <IP>’, you will be prompted for credentials. Default on the Platform Emulator is admin/admin, so enter this and you are good to go.

There are hundreds of cmdlets in the UCS PowerTool module, I am not going to go through them here, but I will show a couple of tactics for, firstly finding what you need in the set of cmdlets, and secondly, for letting PowerTool do the heavy lifting, and produce the script for the build for you.

So lets start with looking for commands in the PowerTool module, we can run ‘Get-Command -Module CiscoUcsPS’, which will give us a list of all the commands in the module. This is a great place to start, and in general the cmdlets are named fairly sensibly, although due to the way PowerTool was converted from the Python API model, some of these are pretty long.

Screen Shot 2015-11-07 at 08.22.46

If we want to search for something more specific, we can use ‘Get-Command -Module CiscoUcsPS *blade*’, for example, to search for all commands in this module with the word ‘blade’ in them. This narrows the search to something you might actually want to do at that time.

Once you have located the cmdlet you want, in this case we will use ‘Get-UcsBlade’ which lists all the blade servers installed in your UCS environment, you can get more information about that by running ‘Get-Help Get-UcsBlade’ followed by one of three optional parameters: ‘-full’, ‘-detailed’ or ‘-examples’. If you run this with no parameters, you will get the basic description and the syntax of the command. If you go for ‘-examples’ you will get examples of the command usage. Entering ‘-detailed’ will go into details on each parameter, what data type they are, whether they are mandatory or optional, basically a tonne of information you probably won’t need (although it’s useful to know this is there when you need it), and ‘-full’ will show you the combination of all three of these.

One thing to be aware of, is that different vendors have different qualities of PowerShell modules. I am a fairly heavy user of the Cisco, VMware, NetApp, and the core Microsoft cmdlets. The VMware ones tend to be very well documented, well presented, and have a tonne of options, while not always providing specific commands which you may need (although using ‘Get-View’ we can tap into anywhere in the API, a story for another day I think). The NetApp commands have excellent examples, but general documentation and community does not really happen, and there are quite a few absent commands which mean you can’t always do what you need to do. The Cisco PowerTool pack has a huge breadth of available commands, which means you can do pretty much anything in there, but they don’t always have examples in the help for the cmdlets, and some of the detailed help is lacking description so leaves success with some commands subject to your trial and error.

Screen Shot 2015-11-07 at 08.23.37

So now you can find the commands you need, and find out how to use them. Well because PowerShell is a declarative tool, we can run single commands from the shell, so if you run ‘Get-UcsBlade’ while connected to your PE, you will see the list of blade servers attached to the system. This list will likely be a few pages so you can trim it down to a table by entering ‘Get-UcsBlade | Format-Table’, or ‘Get-UcsBlade | ft’ for short, which will present the same output with only a few select fields in a nice small table. There are a load of ways of playing with this output to customise it for what you need. i’m not going to go into that here, but suffice to say this is a good way to get information out of your UCS system.

By using the commands, you can build your UCS system piece by piece, this is going to take you a while as you get used to the nuances of PowerTool, and if you have the patience for this then great, you will be far stronger at the end of it, but when I started I used a different method which I will now describe.

One of the great tools we have in PowerTool, is a cmdlet called ‘ConvertTo-UcsCmdlet’. This is a life saver, as it lets you automate, without really needing to know how to write PowerShell scripts. So it works like this.

Once you are connected to your PE through PowerTool (using Connect-Ucs <IP>), you enter the ‘ConvertTo-UcsCmdlet’ command, and the command prompt will disappear. If you need to get out of this mode, just press Ctrl+C, but just leave it for now.

Open up your UCS Manager through your PE’s web front end, and log in. Now go and create something, say a Network Control Policy, for something simple, click OK to save your new policy, and go back to your PowerShell window.

You should see what our magical cmdlet has done, it should have dumped the PowerTool commands for what you just did in the window. Now you can just copy and paste this cmdlet into your favourite text editor, and voila, you don’t need to click buttons to do that again, you can just use this piece of script.

Through doing this, you can build up the full configuration in a text file, name the file with a .ps1 extension, and when you are ready to test it you can factory reset the PE through the web interface and run it again. In a few hours you can quickly create the full build from start to finish using PowerTool.

There are some things which ConvertTo-UcsCmdlet will not convert, creation of Service Profiles is one, but if you look around online there are plenty of good people sharing scripts which can be modified for your purposes.

Hope this helps people, it certainly changed things for me, taking a 3 hour build down to around 30 seconds. Once you have this up and running, you can take what you have and replace the specific elements, say sub-organisation name, and replace them with variables, this script can then be reused again and again. This is a quick way to open your eyes to the power of scripting.

PowerCLI – where to start

I began using PowerShell around 18 months ago while working for a small UK based Managed Service provider. Prior to this, my coding/scripting experience consisted of an A-Level in Computing, which introduced me to Visual Basic 6.0 and databases, a void of around 7 years, and then some sysadmin VBScript and batch file type goodness for a few years.

Screen Shot 2015-11-02 at 21.34.14

Until I started at said company, I had only been exposed to systems running Windows Server 2003, and with a look to security über alles, no access to PowerShell, or any other exciting languages was available, so VBScript became our automation tool of choice.

I have posted before about good resources to use to learn PowerShell, this is more a rundown of how I learned, and the joy and knowledge it gave me to do this.

My first taste of PowerShell was working with Exchange 2010 servers, doing stuff like this to report on mailbox items over a certain age.

Get-Mailbox "username" | New-MailboxSearch -Name search123 -SearchQuery "Received:<01/01/2014" -estimateonly

Were it not for the necessity to use PowerShell to do anything remotely useful in Exchange 2010, I would have been happy to continue to use batch files and VBScript to automate some of the things, I was confident in using these tools, and could achieve time savings, albeit fairly slowly. But PowerShell I must, so PowerShell I did.

Around this time, I became more keen on working with infrastructure, than applications, and got transferred to a role solely looking after our fairly sizeable Cisco UCS and VMware estate. I had plenty of years of experience of VMware, and none of Cisco UCS, but was excited by the new challenge.

I was quickly steered by the senior engineers, towards Cisco PowerTool, and VMware’s PowerCLI, to help to automate some of the administrative, and reporting type tasks I would soon be inundated with, so I picked them up and learned as I went.

I started small, and Google was my friend. Scripting small tasks to save incrementally larger amounts of time. Stuff like this:

$podcsv = import-csv .\UCS_Pods.csv
$credcsv = import-csv .\UCS_Credentials.csv
$ucsuser = $credcsv.username
$ucspasswd = $credcsv.password
$secpasswd = convertto-securestring $ucspasswd -asplaintext -force
$ucscreds = new-object ($ucsuser,$secpasswd)
$datetime = get-date -uformat “%C%y%m%d-%H%M”
foreach($pod in $podcsv)
connect-ucs -credential $ucscreds $podip
get-ucsfault | select ucs,id,lasttransition,descr,ack,severity | export-csv -path .\$datetime-$podname-errors.csv

To dump out the alerts we had in multiple UCS systems, to CSV files. This would save 20-30 minutes a day, nothing major, but clicking buttons is boring, and I can always find better things to do with my time.

On the VMware side of things, I started really small, with stuff like this which would tell you the version of VMTools on all of your virtual machines:

# Ask for connection details, then connect using these
$vcenter = Read-Host "Enter vCenter Name or IP"
$username = Read-Host "Enter your username"
$password = Read-Host "Enter your password"
# Set up our constants for logging
$datetime = get-date -uformat "%C%y%m%d-%H%M"
$outfilepsp = $(".\" + $datetime + "_" + $vcenter + "_PSPList_Log.txt")
$outfilerdm = $(".\" + $datetime + "_" + $vcenter + "_RDMList_Log.txt")
$OutputFile = ".\" + $datetime + "_" + $vcenter + "_VMTools_Report.txt"
# Connect to vCenter
$Connection = Connect-VIServer $vcenter #-User $username -Password $password
foreach($Cluster in Get-Cluster) {
foreach($esxhost in ($Cluster | Get-VMHost | Where { ($_.ConnectionState -eq "Connected") -or ($_.ConnectionState -eq "Maintenance")} | Sort Name)) {
Get-Cluster | Get-VMhost $esxhost | get-vm | % { get-view $ } | select Name, @{ Name="ToolsVersion"; Expression={$}}, @{ Name="ToolStatus"; Expression={$_.Guest.ToolsVersionStatus}}, @{Name="Host";Expression={$esxhost}}, @{Name="Cluster";Expression={$}} | Format-Table | Out-File -FilePath $OutputFile -Append
Disconnect-VIServer * -Confirm:$false

This is a real time saver, and great for getting quick figures out of your environment. As I wrote these scripts, I learned more and more what I could do, picking up ways of doing different things here and there: for/next loops, do/while loops, arrays. As I picked up these concepts again, concepts I had learned years earlier and not used to great effect, my scripts became more complex, and delivered more value in the output they gave, and the time saved. Scripts like this which reports on any datastores over 90% utilisation, these soon became a part of our daily reporting regime:

$datetime = get-date -uformat "%C%y%m%d-%H%M"
$vcentercsv = import-csv .\VCenter_Servers.csv
# Configure connection settings using Read Only account
$credcsv = import-csv .\VMware_Credentials.csv
$vmuser = $credcsv.username
$vmpasswd = $credcsv.password
$secpasswd = convertto-securestring $vmpasswd -asplaintext -force
$vmcreds = new-object ($vmuser,$secpasswd)
$report = @()
foreach($vcenter in $vcentercsv)
connect-viserver $vcenter.ip -credential $vmcreds
foreach ($datastore in (get-datastore | where {$ -notlike "*local*" -and [math]::Round(100-($_.freespacegb/$_.capacitygb)*100) -gt 90}))
$row = '' | select Name,FreeSpaceGB,CapacityGB,vCenter,PercentUsed
$row.Name = $
$row.FreeSpaceGB = $datastore.freespacegb
$row.CapacityGB = $datastore.capacitygb
$row.vCenter = $
$row.PercentUsed = [math]::Round(100-($datastore.freespacegb/$datastore.capacitygb)*100)
$report += $row
Disconnect-VIServer * -Confirm:$false
$report | Sort PercentUsed | export-csv -path .\$datetime-datastore-overuse.csv

My knowledge of how to do things, and confidence in what I was doing grew rapidly, and the old thing of ‘the more I know, the more I realise I don’t know’ came to pass. I am still learning at a rapid rate how better to put these things together, and new cmdlets, new modules, new ways to do things. It’s a fun journey though, one which leaves you with extremely useful and admired skills, and one which will continue to develop you as an IT technician throughout your career.

I am now doing the biggest PowerShell datacenter automation project I have ever done, it is around 5000 lines now, and growing every day. I feel like anything can be achieved with PowerShell, and the various modules released by vendors, and finding ways of solving the constant puzzles which hit me in the face is exciting and rewarding in equal measure.

Everywhere you look in IT now, it is automation and DevOps. It has been said many times that IT engineers who do not learn some form of automation are going to be automated out of a job, and to some extent I agree with this. The advent of software defined storage, networking, everything, shows that automation, and policy driven configuration, is really changing the world of IT infrastructure. If you’re in IT then you probably got in because you love technology, well get out there and learn new skills, whatever those may be, you will enjoy it more than you think.

FlexPod 101 – What is a FlexPod?

I haven’t posted for a while, I started a new job, getting out of IT support, and into the area I want to be, designing and implementing infrastructure solutions as a FlexPod consultant. I have not worked with FlexPod as a concept before, but I have worked with the integral technology stack which comprises it. So far so good, it seems like a robust solution which provides a balance between scalability, performance and cost. I have decided to do a set of blog posts going through the concept and technology behind FlexPod, hopefully highlighting what sets this apart from the competition.

FlexPod: what the hell is that?

Over the last few years, the IT industry has moved from disparate silos for storage, compute and network, towards the converged (and later hyper converged) dream. One such player in this market is the FlexPod.

A collaboration between NetApp and Cisco, at a basic level this comprises the following enterprise class hardware:

  • Cisco Unified Computing System (UCS)
  • Cisco Nexus switching/routing
  • NetApp FAS Storage Arrays

This forms the hardware basis, and as is the industry’s want, there are a swathe of virtualisation solutions and business critical applications which can be used on top of the hardware:

  • VMware vSphere
  • Microsoft Hyper-V
  • Openstack
  • Citrix XenDesktop
  • SAP
  • Oracle
  • VMware Horizon View

I have been a fan of NetApp storage, and Cisco UCS compute for a while. They both offer simplicity and power in equal measure. My preference for hypervisor is ESXi but Hyper-V is becoming a more compelling solution with every release.

You throw an automation product like vRealize Automation or UCS Director on top of the stack and you have a powerful and modern private cloud solution which takes you beyond what a standard virtualised solution will deliver.

Why not just by <enter converged/hyper-converged vendor here>?

But you can run this on any hardware, right? So what sets FlexPod apart from VCE’s Vblock, hyper-converged systems like Nutanix, SimpliVity, or just rolling your own infrastructure?

The answer is the Cisco Validated Design (CVD). This is, as the name suggests, a validated and documented build blueprint, which details proven hardware and software configurations for a growing number of solutions. This gives a confidence when implementing the solution, that this will work, and goes towards putting the ‘Flex’ in FlexPod.

The other advantage of FlexPod over other converged/hyper-converged solutions is that you can tweak the scales on the hardware components (conpute/storage/network), to make the solution larger in the areas you need capacity boost, but keep it the same in the areas you don’t. You need 100TB of storage, just buy more shelves. You need 100 hosts, just buy more UCS chassis and blades. This non-linear scalability, and flexibility, separates FlexPod from rival solutions.

As far as the software, and general protocol usage goes, FlexPod is fairly agnostic. You can use FCoE, NFS, FC or iSCSI as your storage protocols, you can use whatever hypervisor you want as long as there is a CVD for it, and chances are you can find one to suit your use case.

Where can I find more information on FlexPod?

The NetApp and Cisco sites have information about what a FlexPod consists of:

The Cisco site also has links to the CVDs, these give a good overview of what the FlexPod is about.

What’s next?

Part 2 of my FlexPod 101 series will go over the physical components of a FlexPod.