Replacing the ‘All Services’ Icon in vRealize Automation

I had a conversation with Ricky El-Qasem (@rickyelqasem) on Twitter this week about the ‘All Services’ logo in vRealize Automation, and whether this could be replaced programatically.

For those which don’t know the pain of this particular element of vRA; when browsing the service catalog, groups of services are listed down the left hand side of the page with icons next to them:

Screen Shot 2017-03-17 at 17.52.42.png

These can all be changed, but until recently the top icon would remain as a blue lego brick, which can make the otherwise slick portal look unsightly. This is shown on the image below:

Screen Shot 2017-03-17 at 17.56.25.png

Now luckily, from vRA 7.1, this has been replaceable through the API, and steps have been documented in the accompanying guide here. This uses the REST API, and means you need to convert the image in PNG into Base-64 encoding in order to push it to the API, a little to manual for me!

So I quickly threw vRA 7.2 up in my home lab and got to work. I chose to script it using Python because I found that I could easily convert the image to Base-64, and I knew I could do the REST calls using the excellent ‘requests’ Python package (info available here). The code I used is available on my GitHub, and is shown below. I also created a script to delete the custom icon, and return things to vanilla state, you know, just in case 😉

Anyway, I hope this is useful for people who want to quickly and easily replace the icon.

#!/usr/bin/env python
# required packages, install with pip if not present
import requests
import json
# disable self-signed cert warnings
# replace these variables
filename = 'service.png'
vra_ip = ''
vra_user = 'administrator@vsphere.local'
vra_pass = 'VMware1!'
vra_tenant = 'vsphere.local'
# don't replace anything from here
# open file and encode it in b64
with open("./"+filename, "rb") as f:
    data =
    encoded = data.encode("base64")
encoded = encoded.replace("\r","")
encoded = encoded.replace("\n","")
# get our authorization token
uri = 'https://'+vra_ip+'/identity/api/tokens'
headers = {'Accept':'application/json','Content-Type':'application/json'}
payload = '{"username":"'+vra_user+'","password":"'+vra_pass+'","tenant":"'+vra_tenant+'"}'
r =, headers=headers, verify=False, data=payload)
token = 'Bearer '+str(json.loads(r.text)["id"])
# send the new icon to the API
uri = 'https://'+vra_ip+'/catalog-service/api/icons'
headers = {'Accept':'application/json','Content-Type':'application/json','Authorization':token}
payload = '{"id":"cafe_default_icon_genericAllServices","fileName":"'+filename+'","contentType":"image/png","image":"'+encoded+'"}'
r =, headers=headers, verify=False, data=payload)
if r.status_code == 201:
    print "Replacement successful"
    print "Expected return code 201, got "+r.status_code+" something went wrong"

Slipstreaming VMXNET3 drivers into Windows builds


A question came up on Twitter this week about pre-loading drivers into Windows builds to allow future devices to be ready to use. I have been doing this for a while, but take it for granted I guess. I was prompted on responding to blog this, which totally makes sense, so here goes.

In my case we are slipstreaming the VMXNET3 driver, which is needed if you want to provision Windows Server VMs on vCenter with the VMXNET3 network adapter, which is required if you don’t want to limit your Windows VMs to the 1Gbps available to the E1000 adapter type.

Here we are doing it using the autoUnattend.xml file, which I use on a virtual floppy disk to automate Windows builds, but this could apply to however you build your servers, and want to install drivers.

So the following command is in my autoUnattend.xml file, near the end, where we do commands to run at first logon:


This uses the pnputil.exe, which is a built in utility to install drivers, and points to the .inf file, sitting on the virtual floppy drive, in a folder with the below files:


As a heads up, I got these files from a VM which already had VMware Tools installed, from the VMXNET3 specific version folder in ‘C:\Windows\System32\DriverStore\FileRepository’.

Hopefully one day this will be included with Windows, but not sure if this is something Microsoft want to do or not, if not then this is the way I do it for now. I don’t think I entirely figured this out on my own, but have been doing it this way for a while, so apologies for not referencing anyone’s blog I poached this from. I’m sure if you are using SCCM or whatever then you could use a similar method to this, with the command line tool used above to do this (and with any driver, not just VMXNET3).

vSphere HTML5 Client Fling Deployment Script

So yesterday, VMware released the HTML5 vSphere Client as a fling, this is available for download here. I have put together a PowerShell script to deploy this to your vSphere environment.

It seems unusual for this to take the form of an OVA, but at least this means that it does not touch your existing vCenter, so should be deployable with less apprehension.

The client itself is issued an IP address from an IP Pool, and therefore has a different IP address to access from vCenter. Deployment of the OVA is pretty straight forward, and instructions for use and setup are in the link above.

There are already a tonne of posts around the features present, and not present, in the vSphere HTML5  Client, I am not going to go over that here, suffice to say, it is a fling for a reason.

This script (at first release) assumes that a valid, enabled IP Pool already exists in vCenter for the IP you allocate to the VM, I will add functionality in the next release to create an IP Pool if one is not already present.

Other than that, you should just need to replace the variables at the top of the script to use it for deployment. The script is available on my GitHub repository at this link.


PowerShell – Could not create SSL/TLS secure channel

I have spent a considerable amount of time in my life battling with the above error message when running PowerShell scripts. Long and short of it is that this can be caused by a few things, but most of the times I have experienced it, the reason is that the endpoint you are trying to connect to is using self-signed certificates, which causes the Invoke-WebRequest, and Invoke-RestMethod commands to throw an error stating:

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

If you hit this, you will know as your web request via standard REST methods will simply refuse to give you anything back.

I had a bunch of scripts written to do automation of the configuration of vRealize Orchestrator, and vRealize Automation 7.0, and these had been heavily tested, and confirmed as working. The way of avoiding the above error is to use the following PowerShell function:

function Ignore-SelfSignedCerts
Write-Host "Adding TrustAllCertsPolicy type." -ForegroundColor White
Add-Type -TypeDefinition @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem)
return true;
Write-Host "TrustAllCertsPolicy type added." -ForegroundColor White
Write-Host $_ -ForegroundColor "Yellow"
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

So not a great start to my Sunday when I found that my scripts no longer worked after a fresh install of the recently released vRealize Orchestrator and vRealize Automation 7.0.1.

After much messing about, I worked out the cause of this, which is that SSLv3 and TLSv1.0 were both disabled in the new releases, as a result we need to either:

a) Enable SSLv3 or TLSv1.0 – probably not the best idea, these have been disabled due to the growing number of security risks in these protocols, and will (presumably) continue to be disabled for every new version of the products going forward

b) Change the way we issue requests, to use TLSv1.2 – this is the way to do it in my opinion, and the code to do this is a simple one-liner:

[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12;

So if you  hit this problem (and if you are a PowerShell scripter, and interacting with REST APIs with your scripts, then you probably will!), then this is how to fix the issue.

FlexPod and UCS – where are we now?

I have been working on FlexPod for about a year now, and on UCS for a couple of years; in my opinion it’s great tech and takes a lot of the pain away from designing new data center infrastructure solutions, taking away the guesswork, and bringing inherent simplicity, reliability and resilience to the space. Over the last year or so, there has been a shift in the Cisco Validated Designs (CVDs) coming out of Cisco, and as things have moved forward, there is a noticeable shift in the way things are going with FlexPod. I should note that some of what I discuss here is already clear, and some is mere conjecture. I think the direction things are going in with FlexPod is a symptom of changes in the industry, but it is clear that converged solutions are becoming more and more desirable as the Enterprise technology world moves forward.

So what are the key changes we are seeing?

The SDN elephant in the room 

Cisco’s ACI (Application-centric Infrastructure) has taken a while to get moving; the replacement of existing 3-tier network architecture with a leaf-spine network is something which is not going to happen overnight. The switched fabric is arguably a great solution for modern data centers, where east-west traffic is the bulk of network activity, and Spanning Tree Protocol continues to be the bane of network admins, but its implementation often requires either green-field deployments, or forklift replacement of large portions of the existing core networking of a data center.

That’s not to say ACI is not doing OK in terms of sales; Cisco’s figures, and case studies, seem to show that there is uptake, and some large customers taking this on. So how does this fit in with FlexPod? Well, the Cisco Validated Designs (CVDs) released over the last 12 months have all included the new Nexus 9000 series switches, rather than the previous stalwart Nexus 5000 series. These are ACI based switches, which are also able to operate in the ‘legacy’ NX-OS mode. Capability wise, in the context of FlexPod, there is not a lot of difference, they now have FCoE support, can do vPC, QoS, Layer 3, and all the good stuff we come to expect from Nexus switches.

So from what I can gather, the inclusion of 9K switches in the FlexPod line (outside of the FlexPod with ACI designs), is there to enable FlexPod customers to more easily move into the leaf/spine ACI network architecture at a later date, should they wish to do this. This makes sense, and the pricing on the 9Ks being used looks favourable over the 5Ks, so this is a win-win for customers, even if they don’t eventually decide to go with ACI.

40GbE standard 

Recent announcements around the Gen 3 UCS Fabric Interconnects have revealed that 40GbE is now going to be the standard for UCS connectivity solutions, and the new chassis designs show 4 x 40GbE QSFP connections, totaling 320Gbps total bandwidth per chassis, this is an incredible throughput, and although I can’t see 99% of customers going anywhere near these levels, it does help to strengthen the UCS platform’s use cases for even the most high performance environments, and reduces the requirement for Infiniband type solutions for high throughput environments.

Another interesting point, and following on from the ACI ramblings above, is that the new 6300 series Fabric Interconnects are now based on the Nexus 9300 switching line, rather than the Nexus 5K based 6200 series. This positions them perfectly to act as a leaf in an ACI fabric one day, should this become the eventual outcome of Cisco’s strategy.


With the announcements about the new UCS generation, came the news that from UCS firmware version 3.1, the software for UCS would now be unified for UCS classic, UCS Mini, and the newish M-series systems, this simplifies things for people looking at version numbers and how they relate to the relevance of the release, and means that there should now be relative feature parity across all footprints of UCS systems.

The most exciting, if you have experienced long running pain with Java, is that the new release incorporates the HTML5 interface, which has been seen on UCS Mini since its release. I’m sure this will bring new challenges with it, but for now at least, something fresh to look forward to for those running UCS classic.

FlexPod Mini – now not so mini 


FlexPod Mini is based on the UCS Mini release, which came out around 18 months ago, this is based on the I/O Modules (or FEXs) in the UCS 5108 chassis, being replaced with UCS 6324 pocket sized Fabric Interconnects, ultimately cramming a single chassis of UCS, and the attached network equipment into just 6 rack units. This could be expanded with C-series servers, but the scale for UCS blades, was strictly limited to the 8 blade limit of a standard chassis. With the announcement of the new Fabric Interconnect models came the news of a new ‘QSFP Scalability Port License’, which allows the 40GbE port on the UCS 6324 FI to be utilized with a 1 x QSFP to 4 x SFP+ cable, to add another chassis to the UCS Mini.

Personally I haven’t installed a UCS Mini, but the form factor is a great fit for certain use cases, and the more flexible this is, the more desire there will be to use this solution. For FlexPod, this ultimately means more suitable use cases, particularly in the ROBO (remote office/branch office) type scenario.

What’s Next? 

So with the FlexPod now having new switches, and new UCS hardware, it seems something new from NetApp is next on the list for FlexPod. The FAS8000 series is a couple of years old now, so we will likely see a refresh on this at some point, probably with 40GbE on board, more flash options, and faster CPUs. The recent purchase of SolidFire by NetApp will also quite probably see some kind of SolidFire based FlexPod CVD coming out of the Cisco/NetApp partnership in the near future.

We are also seeing the release of some exciting (or at least as exciting as these things can be!) new software releases this year: Windows Server 2016, vSphere 6.5 (assuming this will be revealed at VMworld), and OpenStack Mitaka, all of which will likely bring new CVDs.

PowerCLI – where to start

I began using PowerShell around 18 months ago while working for a small UK based Managed Service provider. Prior to this, my coding/scripting experience consisted of an A-Level in Computing, which introduced me to Visual Basic 6.0 and databases, a void of around 7 years, and then some sysadmin VBScript and batch file type goodness for a few years.

Screen Shot 2015-11-02 at 21.34.14

Until I started at said company, I had only been exposed to systems running Windows Server 2003, and with a look to security über alles, no access to PowerShell, or any other exciting languages was available, so VBScript became our automation tool of choice.

I have posted before about good resources to use to learn PowerShell, this is more a rundown of how I learned, and the joy and knowledge it gave me to do this.

My first taste of PowerShell was working with Exchange 2010 servers, doing stuff like this to report on mailbox items over a certain age.

Get-Mailbox "username" | New-MailboxSearch -Name search123 -SearchQuery "Received:<01/01/2014" -estimateonly

Were it not for the necessity to use PowerShell to do anything remotely useful in Exchange 2010, I would have been happy to continue to use batch files and VBScript to automate some of the things, I was confident in using these tools, and could achieve time savings, albeit fairly slowly. But PowerShell I must, so PowerShell I did.

Around this time, I became more keen on working with infrastructure, than applications, and got transferred to a role solely looking after our fairly sizeable Cisco UCS and VMware estate. I had plenty of years of experience of VMware, and none of Cisco UCS, but was excited by the new challenge.

I was quickly steered by the senior engineers, towards Cisco PowerTool, and VMware’s PowerCLI, to help to automate some of the administrative, and reporting type tasks I would soon be inundated with, so I picked them up and learned as I went.

I started small, and Google was my friend. Scripting small tasks to save incrementally larger amounts of time. Stuff like this:

$podcsv = import-csv .\UCS_Pods.csv
$credcsv = import-csv .\UCS_Credentials.csv
$ucsuser = $credcsv.username
$ucspasswd = $credcsv.password
$secpasswd = convertto-securestring $ucspasswd -asplaintext -force
$ucscreds = new-object ($ucsuser,$secpasswd)
$datetime = get-date -uformat “%C%y%m%d-%H%M”
foreach($pod in $podcsv)
connect-ucs -credential $ucscreds $podip
get-ucsfault | select ucs,id,lasttransition,descr,ack,severity | export-csv -path .\$datetime-$podname-errors.csv

To dump out the alerts we had in multiple UCS systems, to CSV files. This would save 20-30 minutes a day, nothing major, but clicking buttons is boring, and I can always find better things to do with my time.

On the VMware side of things, I started really small, with stuff like this which would tell you the version of VMTools on all of your virtual machines:

# Ask for connection details, then connect using these
$vcenter = Read-Host "Enter vCenter Name or IP"
$username = Read-Host "Enter your username"
$password = Read-Host "Enter your password"
# Set up our constants for logging
$datetime = get-date -uformat "%C%y%m%d-%H%M"
$outfilepsp = $(".\" + $datetime + "_" + $vcenter + "_PSPList_Log.txt")
$outfilerdm = $(".\" + $datetime + "_" + $vcenter + "_RDMList_Log.txt")
$OutputFile = ".\" + $datetime + "_" + $vcenter + "_VMTools_Report.txt"
# Connect to vCenter
$Connection = Connect-VIServer $vcenter #-User $username -Password $password
foreach($Cluster in Get-Cluster) {
foreach($esxhost in ($Cluster | Get-VMHost | Where { ($_.ConnectionState -eq "Connected") -or ($_.ConnectionState -eq "Maintenance")} | Sort Name)) {
Get-Cluster | Get-VMhost $esxhost | get-vm | % { get-view $ } | select Name, @{ Name="ToolsVersion"; Expression={$}}, @{ Name="ToolStatus"; Expression={$_.Guest.ToolsVersionStatus}}, @{Name="Host";Expression={$esxhost}}, @{Name="Cluster";Expression={$}} | Format-Table | Out-File -FilePath $OutputFile -Append
Disconnect-VIServer * -Confirm:$false

This is a real time saver, and great for getting quick figures out of your environment. As I wrote these scripts, I learned more and more what I could do, picking up ways of doing different things here and there: for/next loops, do/while loops, arrays. As I picked up these concepts again, concepts I had learned years earlier and not used to great effect, my scripts became more complex, and delivered more value in the output they gave, and the time saved. Scripts like this which reports on any datastores over 90% utilisation, these soon became a part of our daily reporting regime:

$datetime = get-date -uformat "%C%y%m%d-%H%M"
$vcentercsv = import-csv .\VCenter_Servers.csv
# Configure connection settings using Read Only account
$credcsv = import-csv .\VMware_Credentials.csv
$vmuser = $credcsv.username
$vmpasswd = $credcsv.password
$secpasswd = convertto-securestring $vmpasswd -asplaintext -force
$vmcreds = new-object ($vmuser,$secpasswd)
$report = @()
foreach($vcenter in $vcentercsv)
connect-viserver $vcenter.ip -credential $vmcreds
foreach ($datastore in (get-datastore | where {$ -notlike "*local*" -and [math]::Round(100-($_.freespacegb/$_.capacitygb)*100) -gt 90}))
$row = '' | select Name,FreeSpaceGB,CapacityGB,vCenter,PercentUsed
$row.Name = $
$row.FreeSpaceGB = $datastore.freespacegb
$row.CapacityGB = $datastore.capacitygb
$row.vCenter = $
$row.PercentUsed = [math]::Round(100-($datastore.freespacegb/$datastore.capacitygb)*100)
$report += $row
Disconnect-VIServer * -Confirm:$false
$report | Sort PercentUsed | export-csv -path .\$datetime-datastore-overuse.csv

My knowledge of how to do things, and confidence in what I was doing grew rapidly, and the old thing of ‘the more I know, the more I realise I don’t know’ came to pass. I am still learning at a rapid rate how better to put these things together, and new cmdlets, new modules, new ways to do things. It’s a fun journey though, one which leaves you with extremely useful and admired skills, and one which will continue to develop you as an IT technician throughout your career.

I am now doing the biggest PowerShell datacenter automation project I have ever done, it is around 5000 lines now, and growing every day. I feel like anything can be achieved with PowerShell, and the various modules released by vendors, and finding ways of solving the constant puzzles which hit me in the face is exciting and rewarding in equal measure.

Everywhere you look in IT now, it is automation and DevOps. It has been said many times that IT engineers who do not learn some form of automation are going to be automated out of a job, and to some extent I agree with this. The advent of software defined storage, networking, everything, shows that automation, and policy driven configuration, is really changing the world of IT infrastructure. If you’re in IT then you probably got in because you love technology, well get out there and learn new skills, whatever those may be, you will enjoy it more than you think.

FlexPod 101 – What is a FlexPod?

I haven’t posted for a while, I started a new job, getting out of IT support, and into the area I want to be, designing and implementing infrastructure solutions as a FlexPod consultant. I have not worked with FlexPod as a concept before, but I have worked with the integral technology stack which comprises it. So far so good, it seems like a robust solution which provides a balance between scalability, performance and cost. I have decided to do a set of blog posts going through the concept and technology behind FlexPod, hopefully highlighting what sets this apart from the competition.

FlexPod: what the hell is that?

Over the last few years, the IT industry has moved from disparate silos for storage, compute and network, towards the converged (and later hyper converged) dream. One such player in this market is the FlexPod.

A collaboration between NetApp and Cisco, at a basic level this comprises the following enterprise class hardware:

  • Cisco Unified Computing System (UCS)
  • Cisco Nexus switching/routing
  • NetApp FAS Storage Arrays

This forms the hardware basis, and as is the industry’s want, there are a swathe of virtualisation solutions and business critical applications which can be used on top of the hardware:

  • VMware vSphere
  • Microsoft Hyper-V
  • Openstack
  • Citrix XenDesktop
  • SAP
  • Oracle
  • VMware Horizon View

I have been a fan of NetApp storage, and Cisco UCS compute for a while. They both offer simplicity and power in equal measure. My preference for hypervisor is ESXi but Hyper-V is becoming a more compelling solution with every release.

You throw an automation product like vRealize Automation or UCS Director on top of the stack and you have a powerful and modern private cloud solution which takes you beyond what a standard virtualised solution will deliver.

Why not just by <enter converged/hyper-converged vendor here>?

But you can run this on any hardware, right? So what sets FlexPod apart from VCE’s Vblock, hyper-converged systems like Nutanix, SimpliVity, or just rolling your own infrastructure?

The answer is the Cisco Validated Design (CVD). This is, as the name suggests, a validated and documented build blueprint, which details proven hardware and software configurations for a growing number of solutions. This gives a confidence when implementing the solution, that this will work, and goes towards putting the ‘Flex’ in FlexPod.

The other advantage of FlexPod over other converged/hyper-converged solutions is that you can tweak the scales on the hardware components (conpute/storage/network), to make the solution larger in the areas you need capacity boost, but keep it the same in the areas you don’t. You need 100TB of storage, just buy more shelves. You need 100 hosts, just buy more UCS chassis and blades. This non-linear scalability, and flexibility, separates FlexPod from rival solutions.

As far as the software, and general protocol usage goes, FlexPod is fairly agnostic. You can use FCoE, NFS, FC or iSCSI as your storage protocols, you can use whatever hypervisor you want as long as there is a CVD for it, and chances are you can find one to suit your use case.

Where can I find more information on FlexPod?

The NetApp and Cisco sites have information about what a FlexPod consists of:

The Cisco site also has links to the CVDs, these give a good overview of what the FlexPod is about.

What’s next?

Part 2 of my FlexPod 101 series will go over the physical components of a FlexPod.