vSphere Automation SDKs

This week VMware open sourced their SDKs for vSphere using REST APIs, and Python. The REST API was released with vSphere 6.0, while the Python SDK has been around for nearly four years now. I’m going to summarise the contents of this release below, and where these can help us make more of our vSphere environments.

REST API

The vSphere REST API has been growing since the release of vSphere 6 nearly two years ago, and brings access to the following areas of vSphere with its current release:

  • Session management
  • Tagging
  • Content Library
  • Virtual Machines
  • vCenter Server Appliance management

These cover mainly new features from vSphere 6.0 (formerly known as vCloud Suite SDK), and then some of the new bits put together for modernising the API access in vSphere 6.5. The Virtual Machine management particularly is useful in being able to start using REST based methods to do operations, and report on VMs in your environment, very useful for people looking to write quick integrations with things like vRealize Orchestrator, where the built in plugins do not do what you want.

The new material, available on GitHub, contains two main functions:

Postman Collection

Screen Shot 2017-03-12 at 10.28.54.png

Postman is a REST client used to explore APIs, providing a nice graphical display of the request-response type methods used for REST. This is a great way to get your head round what is happening with requests, and helps to build up an idea of what is going on with the API.

Pre-built packs of requests can be gathered together in Postman ‘Collections’; these can then be distributed (in JSON format) and loaded into another instance of Postman. This can be crucially important in documenting the functionality of APIs, especially when the documentation is lacking.

There are some instructions on how to set this up here; if you are new to REST APIs, or just want a quick way to have a play with the new vSphere REST APIs, you could do far worse than starting here.

Node.js Sample Pack

Node.js has taken over the world of server side web programming, and thanks to the simple syntax of Javascript, is easy to pick up and get started with. This pack (available here) has some samples of Node.js code to interact with the REST API. This is a good place to start with seeing how web requests and responses are dealt with in Node, and how we can programatically carry out administrative tasks.

These could be integrated into a web based portal to do the requests directly, or I can see these being used in the future as part of a serverless administration platform, using something like AWS Lambda along with a monitoring platform to automate the administration of a vSphere environment.

Python SDK

Python has been an incredibly popular language for automation for a number of years. Its very low barrier to getting started makes it ideal to pick up and learn, with a wealth of possibilities for building on solid simple foundations to make highly complex software solutions. VMware released their ‘pyvmomi’ Python SDK back in 2013, and it has received consistent updates since then. While not as popular, or as promoted as their PowerCLI PowerShell module, it has nevertheless had strong usage and support from the community.

The release on offer as part of the vSphere Automation SDKs consists of scripts to spin up a demo environment for developing with the Python SDK, as well as a number of sample scripts demonstrating the functionality of the new APIs released in vSphere 6.0 and 6.5.

The continued growth in popularity of Python, along with leading automation toolsets like Ansible using a Python base, mean that it is a great platform to push this kind of development and publicity in. As with Node.js; serverless platforms are widely supporting Python, so this could be integrated with Lambda, Fission, or other FaaS platforms in the future.

Conclusion

It’s great to see VMware really getting behind developing and pushing their automation toolkits in the open, they are to my mind a leader in the industry in terms of making their products programmable, and I hope they continue at this pace and in this vein. The work shown in this release will help make it easier for people new to automation to get involved and start reaping the benefits that it can bring, and the possibilities for combining these vSphere SDKs with serverless administration will be an interesting area to watch.

Advertisements

Automating UCS System Builds

A project I have been working on recently is the full automation of the Cisco UCS system build. Automation comes from a desire to not sit clicking buttons; that’s a great way to learn how a system fits together, but once you have done it once or twice, it is no longer exciting. A look at the FlexPod Cisco Validated Design (CVD) shows around 50 pages of configuration steps. This is largely the same, regardless of the client, and will take at least 3 hours of clicking buttons. 

This shows this to be a prime candidate for automation. There are a few options available here for the automation of this task:

  • Cisco PowerTool for UCS
  • Python API for Cisco UCS
  • Altering of XML backups and restoration onto the UCS system

Altering an exported XML configuration backup, to fit the customer’s solution works, but it is not particularly neat or nice, and will result in replacing all of the system specific information through copy/paste, or a more neat solution parsing the XML and replacing elements. This is not something I really want to do, and it does not leave us with a particularly customisable solution.

The Python API is extensive, and has a tonne of documentation around it. I have run through the Codecademy course for Python, and understand the basics, but I come from a Windows administrative background, and I am not comfortable enough that i want to do this from scratch for UCS. This is something to put down for the future, as my Python knowledge grows. The great advantage in using Python is that it is platform agnostic, so I could run this from a Mac, Linux, or Windows environment (as long as i have the Python packages installed). Sounds great, but the documentation from Cisco around Managed Objects melts my brain, so this is something i discounted for now.

Luckily, Cisco have done a fantastic job with their PowerShell library for Cisco UCS. This is simple to get started with, and the great thing about scripting with PowerShell is that anyone can read it and figure out what the script is doing. As an infrastructure engineer, with a background in PowerShell and VMware’s equally excellent PowerCLI PowerShell module. this was the natural fit for me.

So where did I, and should you (in my opinion), start with automating your Cisco UCS builds?

The first element of this i have already mentioned, get Cisco PowerTool. This is available for download from Cisco’s site, and once you have the MSI installed, you can launch your PowerShell console from Windows, and run ‘Import-Module CiscoUCSPS’ to import the module. You now have the power at your fingertips.

The next good place to start, whether you know a lot about UCS or not, is to get the UCS Platform Emulator, and get this deployed. This comes as an OVA appliance, which you can deploy to VMware Player, VMware Workstation, ESXi, or any other compatible Type 1 or Type 2 hypervisor. Once this has booted, go to the console and give it an IP (if it didn’t already get one from your DHCP server).

It should be noted that there are currently 3 active versions of the UCSPE available: one for UCS standard, one for UCS M-series servers, and one for UCS Mini. Make sure you get the right one for what you are wanting to automate. In this example we are looking at standard UCS, so grab that one, and spin it up.

Now we have the PE up and IP’d, so open your web browser and go to the IP. Here you can amend the hardware configuration to match your environment if that is useful. Most of the automation you will be doing is creating policies, configuring the Fabric Interconnects, creating service profiles, so the exact hardware configuration is not hugely important, and customising the hardware in here can be quite time consuming and frustrating as the interface is clunky and unintuitive.

So now we have our Platform Emulator stood up, we can connect in to it using PowerTool. Open a PowerTool window and enter ‘Connect-Ucs <IP>’, you will be prompted for credentials. Default on the Platform Emulator is admin/admin, so enter this and you are good to go.

There are hundreds of cmdlets in the UCS PowerTool module, I am not going to go through them here, but I will show a couple of tactics for, firstly finding what you need in the set of cmdlets, and secondly, for letting PowerTool do the heavy lifting, and produce the script for the build for you.

So lets start with looking for commands in the PowerTool module, we can run ‘Get-Command -Module CiscoUcsPS’, which will give us a list of all the commands in the module. This is a great place to start, and in general the cmdlets are named fairly sensibly, although due to the way PowerTool was converted from the Python API model, some of these are pretty long.

Screen Shot 2015-11-07 at 08.22.46

If we want to search for something more specific, we can use ‘Get-Command -Module CiscoUcsPS *blade*’, for example, to search for all commands in this module with the word ‘blade’ in them. This narrows the search to something you might actually want to do at that time.

Once you have located the cmdlet you want, in this case we will use ‘Get-UcsBlade’ which lists all the blade servers installed in your UCS environment, you can get more information about that by running ‘Get-Help Get-UcsBlade’ followed by one of three optional parameters: ‘-full’, ‘-detailed’ or ‘-examples’. If you run this with no parameters, you will get the basic description and the syntax of the command. If you go for ‘-examples’ you will get examples of the command usage. Entering ‘-detailed’ will go into details on each parameter, what data type they are, whether they are mandatory or optional, basically a tonne of information you probably won’t need (although it’s useful to know this is there when you need it), and ‘-full’ will show you the combination of all three of these.

One thing to be aware of, is that different vendors have different qualities of PowerShell modules. I am a fairly heavy user of the Cisco, VMware, NetApp, and the core Microsoft cmdlets. The VMware ones tend to be very well documented, well presented, and have a tonne of options, while not always providing specific commands which you may need (although using ‘Get-View’ we can tap into anywhere in the API, a story for another day I think). The NetApp commands have excellent examples, but general documentation and community does not really happen, and there are quite a few absent commands which mean you can’t always do what you need to do. The Cisco PowerTool pack has a huge breadth of available commands, which means you can do pretty much anything in there, but they don’t always have examples in the help for the cmdlets, and some of the detailed help is lacking description so leaves success with some commands subject to your trial and error.

Screen Shot 2015-11-07 at 08.23.37

So now you can find the commands you need, and find out how to use them. Well because PowerShell is a declarative tool, we can run single commands from the shell, so if you run ‘Get-UcsBlade’ while connected to your PE, you will see the list of blade servers attached to the system. This list will likely be a few pages so you can trim it down to a table by entering ‘Get-UcsBlade | Format-Table’, or ‘Get-UcsBlade | ft’ for short, which will present the same output with only a few select fields in a nice small table. There are a load of ways of playing with this output to customise it for what you need. i’m not going to go into that here, but suffice to say this is a good way to get information out of your UCS system.

By using the commands, you can build your UCS system piece by piece, this is going to take you a while as you get used to the nuances of PowerTool, and if you have the patience for this then great, you will be far stronger at the end of it, but when I started I used a different method which I will now describe.

One of the great tools we have in PowerTool, is a cmdlet called ‘ConvertTo-UcsCmdlet’. This is a life saver, as it lets you automate, without really needing to know how to write PowerShell scripts. So it works like this.

Once you are connected to your PE through PowerTool (using Connect-Ucs <IP>), you enter the ‘ConvertTo-UcsCmdlet’ command, and the command prompt will disappear. If you need to get out of this mode, just press Ctrl+C, but just leave it for now.

Open up your UCS Manager through your PE’s web front end, and log in. Now go and create something, say a Network Control Policy, for something simple, click OK to save your new policy, and go back to your PowerShell window.

You should see what our magical cmdlet has done, it should have dumped the PowerTool commands for what you just did in the window. Now you can just copy and paste this cmdlet into your favourite text editor, and voila, you don’t need to click buttons to do that again, you can just use this piece of script.

Through doing this, you can build up the full configuration in a text file, name the file with a .ps1 extension, and when you are ready to test it you can factory reset the PE through the web interface and run it again. In a few hours you can quickly create the full build from start to finish using PowerTool.

There are some things which ConvertTo-UcsCmdlet will not convert, creation of Service Profiles is one, but if you look around online there are plenty of good people sharing scripts which can be modified for your purposes.

Hope this helps people, it certainly changed things for me, taking a 3 hour build down to around 30 seconds. Once you have this up and running, you can take what you have and replace the specific elements, say sub-organisation name, and replace them with variables, this script can then be reused again and again. This is a quick way to open your eyes to the power of scripting.

PowerCLI – where to start

I began using PowerShell around 18 months ago while working for a small UK based Managed Service provider. Prior to this, my coding/scripting experience consisted of an A-Level in Computing, which introduced me to Visual Basic 6.0 and databases, a void of around 7 years, and then some sysadmin VBScript and batch file type goodness for a few years.

Screen Shot 2015-11-02 at 21.34.14

Until I started at said company, I had only been exposed to systems running Windows Server 2003, and with a look to security über alles, no access to PowerShell, or any other exciting languages was available, so VBScript became our automation tool of choice.

I have posted before about good resources to use to learn PowerShell, this is more a rundown of how I learned, and the joy and knowledge it gave me to do this.

My first taste of PowerShell was working with Exchange 2010 servers, doing stuff like this to report on mailbox items over a certain age.

Get-Mailbox "username" | New-MailboxSearch -Name search123 -SearchQuery "Received:<01/01/2014" -estimateonly

Were it not for the necessity to use PowerShell to do anything remotely useful in Exchange 2010, I would have been happy to continue to use batch files and VBScript to automate some of the things, I was confident in using these tools, and could achieve time savings, albeit fairly slowly. But PowerShell I must, so PowerShell I did.

Around this time, I became more keen on working with infrastructure, than applications, and got transferred to a role solely looking after our fairly sizeable Cisco UCS and VMware estate. I had plenty of years of experience of VMware, and none of Cisco UCS, but was excited by the new challenge.

I was quickly steered by the senior engineers, towards Cisco PowerTool, and VMware’s PowerCLI, to help to automate some of the administrative, and reporting type tasks I would soon be inundated with, so I picked them up and learned as I went.

I started small, and Google was my friend. Scripting small tasks to save incrementally larger amounts of time. Stuff like this:


$podcsv = import-csv .\UCS_Pods.csv
$credcsv = import-csv .\UCS_Credentials.csv
$ucsuser = $credcsv.username
$ucspasswd = $credcsv.password
$secpasswd = convertto-securestring $ucspasswd -asplaintext -force
$ucscreds = new-object system.management.automation.pscredential ($ucsuser,$secpasswd)
$datetime = get-date -uformat “%C%y%m%d-%H%M”
foreach($pod in $podcsv)
{
$podname=$pod.name
$podip=$pod.ip
connect-ucs -credential $ucscreds $podip
get-ucsfault | select ucs,id,lasttransition,descr,ack,severity | export-csv -path .\$datetime-$podname-errors.csv
disconnect-ucs
}

To dump out the alerts we had in multiple UCS systems, to CSV files. This would save 20-30 minutes a day, nothing major, but clicking buttons is boring, and I can always find better things to do with my time.

On the VMware side of things, I started really small, with stuff like this which would tell you the version of VMTools on all of your virtual machines:


# Ask for connection details, then connect using these
$vcenter = Read-Host "Enter vCenter Name or IP"
$username = Read-Host "Enter your username"
$password = Read-Host "Enter your password"
# Set up our constants for logging
$datetime = get-date -uformat "%C%y%m%d-%H%M"
$outfilepsp = $(".\" + $datetime + "_" + $vcenter + "_PSPList_Log.txt")
$outfilerdm = $(".\" + $datetime + "_" + $vcenter + "_RDMList_Log.txt")
$OutputFile = ".\" + $datetime + "_" + $vcenter + "_VMTools_Report.txt"
# Connect to vCenter
$Connection = Connect-VIServer $vcenter #-User $username -Password $password
foreach($Cluster in Get-Cluster) {
foreach($esxhost in ($Cluster | Get-VMHost | Where { ($_.ConnectionState -eq "Connected") -or ($_.ConnectionState -eq "Maintenance")} | Sort Name)) {
Get-Cluster | Get-VMhost $esxhost | get-vm | % { get-view $_.id } | select Name, @{ Name="ToolsVersion"; Expression={$_.config.tools.toolsVersion}}, @{ Name="ToolStatus"; Expression={$_.Guest.ToolsVersionStatus}}, @{Name="Host";Expression={$esxhost}}, @{Name="Cluster";Expression={$cluster.name}} | Format-Table | Out-File -FilePath $OutputFile -Append
}
}
Disconnect-VIServer * -Confirm:$false

This is a real time saver, and great for getting quick figures out of your environment. As I wrote these scripts, I learned more and more what I could do, picking up ways of doing different things here and there: for/next loops, do/while loops, arrays. As I picked up these concepts again, concepts I had learned years earlier and not used to great effect, my scripts became more complex, and delivered more value in the output they gave, and the time saved. Scripts like this which reports on any datastores over 90% utilisation, these soon became a part of our daily reporting regime:


$datetime = get-date -uformat "%C%y%m%d-%H%M"
$vcentercsv = import-csv .\VCenter_Servers.csv
# Configure connection settings using Read Only account
$credcsv = import-csv .\VMware_Credentials.csv
$vmuser = $credcsv.username
$vmpasswd = $credcsv.password
$secpasswd = convertto-securestring $vmpasswd -asplaintext -force
$vmcreds = new-object system.management.automation.pscredential ($vmuser,$secpasswd)
$report = @()
foreach($vcenter in $vcentercsv)
{
$vcentername=$vcenter.name
connect-viserver $vcenter.ip -credential $vmcreds
foreach ($datastore in (get-datastore | where {$_.name -notlike "*local*" -and [math]::Round(100-($_.freespacegb/$_.capacitygb)*100) -gt 90}))
{
$row = '' | select Name,FreeSpaceGB,CapacityGB,vCenter,PercentUsed
$row.Name = $datastore.name
$row.FreeSpaceGB = $datastore.freespacegb
$row.CapacityGB = $datastore.capacitygb
$row.vCenter = $vcenter.name
$row.PercentUsed = [math]::Round(100-($datastore.freespacegb/$datastore.capacitygb)*100)
$report += $row
}
Disconnect-VIServer * -Confirm:$false
}
$report | Sort PercentUsed | export-csv -path .\$datetime-datastore-overuse.csv

My knowledge of how to do things, and confidence in what I was doing grew rapidly, and the old thing of ‘the more I know, the more I realise I don’t know’ came to pass. I am still learning at a rapid rate how better to put these things together, and new cmdlets, new modules, new ways to do things. It’s a fun journey though, one which leaves you with extremely useful and admired skills, and one which will continue to develop you as an IT technician throughout your career.

I am now doing the biggest PowerShell datacenter automation project I have ever done, it is around 5000 lines now, and growing every day. I feel like anything can be achieved with PowerShell, and the various modules released by vendors, and finding ways of solving the constant puzzles which hit me in the face is exciting and rewarding in equal measure.

Everywhere you look in IT now, it is automation and DevOps. It has been said many times that IT engineers who do not learn some form of automation are going to be automated out of a job, and to some extent I agree with this. The advent of software defined storage, networking, everything, shows that automation, and policy driven configuration, is really changing the world of IT infrastructure. If you’re in IT then you probably got in because you love technology, well get out there and learn new skills, whatever those may be, you will enjoy it more than you think.