28 August 2012

Even Faster PowerCLI Code with Get-View, UpdateViewData() and LinkedViews

We are always looking for speed, speed, speed in our PowerShell / PowerCLI scripts.  When properly used, the Get-View cmdlet is known to be far faster than using other PowerCLI cmdlets for getting info (such as Get-VM, Get-VMHost, Get-Datastore, etc.).  It can take quite a bit longer to write a script using Get-View versus using other cmdlets, but the speed payoff is usually well worth the additional upfront time investement.  VMware has provided a way to speed things up even more.

A while back (Aug 2011), the official PowerCLI blog had a post about how to "Optimize the performance of PowerCLI's views".  The technique is centered around using the UpdateViewData() method of a "view" of a managed object (".NET View object") to retrieve nested views of managed objects.

The UpdateViewData() method had already been able to selectively update properties of the given managed object view, but one would have to use Get-View again to retrieve views of nested managed objects that are properties of that parent managed object.  "So, what?"  So, speed!

Example 0:  Get guest IP info for VMs using given VM network

Say you wanted to get the IPs configured on VMs that are connected to particular VM networks.  This example goes about doing so by getting said network(s), getting the VMs connected to them, and then getting guest IP info.

You could use the following standard way using multiple Get-View calls:
## the network name pattern to use for getting vitual networks
$strNetworkNamePattern = "MyVMNetwork1711"
## get the virtual networks whose name match this pattern
$arrNetworkViews = Get-View -ViewType Network -Property Name,VM -Filter @{"Name" = $strNetworkNamePattern}

## get the VM managed objects with a couple of properties
$arrVMsUsingThis = $arrNetworkViews | ?{$_.Vm -ne $null} | %{Get-View -Property Name,Guest.Net $_.Vm}

## get info about IPs used on this/these network(s)
$arrVMsUsingThis | %{
   $strVMName = $_.Name
   ## for each Guest.Net where the network name is like the given network name
   $_.Guest.Net | ?{$_.Network -like "*$strNetworkNamePattern*"} | %{
       ## get IPs
       $_.IpAddress | %{
           New-Object -TypeName PSObject -Property @{
               VMName = $strVMName
               IPAddr = $_
           } ## end new-object
       } ## end %
   } ## end %
} ## end %

Or, using the UpdateViewData() method, the code would be:
## the network name pattern to use for getting vitual networks
$strNetworkNamePattern = "MyVMNetwork1711"
## get the virtual networks whose name match this pattern
$arrNetworkViews = Get-View -ViewType Network -Property Name -Filter @{"Name" = $strNetworkNamePattern}

## get a couple of properties for the VMs using this network
$arrNetworkViews | %{$_.UpdateViewData("VM.Name","VM.Guest.Net")}

## get info about IPs used on this/these network(s)
$arrNetworkViews | %{$_.LinkedView.Vm} | %{
   $strVMName = $_.Name
   ## for each Guest.Net where the network name is like the given network name
   $_.Guest.Net | ?{$_.Network -like "*$strNetworkNamePattern*"} | %{
       ## get IPs
       $_.IpAddress | %{
           New-Object -TypeName PSObject -Property @{
               VMName = $strVMName
               IPAddr = $_
           } ## end new-object
       } ## end %
   } ## end %
} ## end %
The only slight differences in the code in the two ways:
  • Line 04 in each:  the first way gets an additional property, "VM", for later use
  • Line 07 in each:  the first way uses and additional Get-View call, whereas the second way uses the UpdateViewData() method
  • Line 10 in each:  both access an array of VMs connected to the given VM network, but getting to those VM arrays is a touch different (the "LinkedView" property in the second way is due to having used the UpdateViewData() method)
This runs reasonably quickly both ways when just getting information for VMs on one VM network.  But, to really illustrate the potential speed increases, I tested the main difference (line 07) with not just one or two networks, but with about 30 networks and with about 160 networks.  The results are impressive.  Again, this was measuring just the central point of each way:  the amount of time for the code on line 07 in each way above to get the VM info from vCenter (first via an additional Get-View, and then via UpdateViewData()).

Get-View vs. UpdateViewData() -- Speed testing results in a couple of different environment sizes:
Method usedNum. networksNum. netwk connectionsTime to run (sec)Total Get-View calls
additional Get-View call3343295.334
UpdateViewData()334327.751
additional Get-View call1593705811.4160
UpdateViewData()159370539.611

Mhmm.  7.75 seconds vs. 95.3 seconds in a small-ish environment -- more than twelve (12) times faster for UpdateViewData().  And 40 seconds vs. 811 seconds in a larger environment -- more than twenty (20) times faster!

Example 1:  Get VMHosts' FirewallSystems to refresh firewall info

Another example.  AC recently had needed to refresh the VMHost firewall info on hosts.  This stemmed from a bug that AC ran into in which the host firewall info is gathered during the host boot process before the firewall services are fully initiated.

Next is the overall code for each way, and then some speed comparisons of each method.  Note, we originally wrote this to only refresh firewall info for VMHosts in one cluster at a time, but to have a bit larger set of operations so as to better illustrate the speed differences, the following code performs the actions for all clusters, one cluster at a time.

Traditional way, using multiple Get-View calls:
## get the Cluster managed objects
Get-View -ViewType ClusterComputeResource -Property Name,Host | %{
   ## get all of this cluster's HostSystem managed objects
   $arrHostViews = Get-View $_.Host -Property ConfigManager.FirewallSystem
   ## for each host in the given cluster, get the FirewallSystem managed object
   foreach ($viewHost in $arrHostViews) {
       $viewFirewallSystem = Get-View $viewHost.ConfigManager.FirewallSystem -Property FirewallInfo.DefaultPolicy
       ## call RefreshFirewall() method for this host's FirewallSystem
       $viewFirewallSystem.RefreshFirewall()
   } ## end %
} ## end %

And, the UpdateViewData() way:
## get the Cluster managed objects
$arrClusterViews = Get-View -ViewType ClusterComputeResource -Property Name
## update the .NET View data with a select FirewallSystem sub-property
$arrClusterViews | %{$_.UpdateViewData("Host.ConfigManager.FirewallSystem.FirewallInfo.DefaultPolicy")}
## for each host in the given cluster, call RefreshFirewall() method
$arrClusterViews | %{$_.LinkedView.Host} | % {$_.ConfigManager.LinkedView.FirewallSystem.RefreshFirewall()}

The differences in code between the two ways:
  • Lines 04-07 in 1st way vs. Line 04 in 2nd way:  The first way uses the standard way of additionally calling Get-View for each cluster to get the HostSystems, and then again for each HostSystem to get the FirewallSystem objects.  The second way uses one UpdateViewData() call to make the FirewallSystem managed objects available from the original array of cluster managed objects.
To see the speed differences for the main info-gathering portion of each way, I measured these two pieces mentioned above:  lines 04-07 for the first way, and line 04 for the second way.  UpdateViewData() wins.

Get-View vs. UpdateViewData() -- Speed testing results in a couple of different environment sizes:
Method usedNum. clustersNum. HostSystemsTime to run (sec)Total Get-View calls
additional Get-View5219.4727
UpdateViewData()5211.391
additional Get-View 3619170.85228
UpdateViewData()361918.201


The speed gain comes, in part, from the sheer number of calls being made that require a round trip request to the given vCenter server(s).  To get additional managed object info using Get-View, there is a Get-View call involved for every [group of] managed object references (MORefs).  Conversely, there is but one (1) call to communicate with vCenter for all "child" managed objects involved when UpdateViewData() is used.

This is not the most obvious route at first, but eventually you can see how to use the method to update managed object view data, and then access the nested managed objects' views using "LinkedView" objects that then reside "in" the parent managed object as properties.

The reward for successfully employing the technique:  speed.  The PowerCLI blog post reports a 100x speed increase in the given example versus using multiple Get-View calls to get nested managed object views.  100x -- that is speeding up a script that takes five (5) minutes to take only three (3) seconds -- FaF!  This speed increase varies based on how many Get-View calls are saved/avoided, what they are retrieving, etc, but that kind of potential speed gain demands attention.

For some further info:  I used this technique in a VMware communities thread about VM portgroup inventory (yes -- December 2011; it has taken me a while to get this post together).  It shows other examples of using the resulting LinkedView properties.  And, there is good info in the PowerCLI lab guide from the 2011 VMworld labs.  The PowerCLI Blog post about PowerCLI Lab at VMWorld 2011 links to the lab manual PDF.  See pages 91-93.

How much faster is your script about to be?

27 August 2012

Power Saving, PSODs, ESXi 5, and Intel E7 CPUs

Strangely enough, I have not found much on the Internet about this particular issue:


Why is that strange?  Primarily because I have a hard time believing we are the only ones using ESXi 5.0 U1 on HP DL580 G7 servers, which appears to be one of the combinations where you would have this issue.  And actually, the problem is apparently more widespread than that.  Rumor is that it affects any server with the Intel Xeon E7 or 75xx CPU in it.  By now you are surely wanting more details.  Read on...

At a client site, we started receiving new HP DL580 G7 servers.  The first use was for VDI and in that environment we weren't (and still aren't) running ESXi 5 yet.  Those happened to have the Intel X7560 CPUs.  Let's ignore that for now.  Later we started deploying new HP DL580 G7 servers with the Intel E7-L8867 in the server environment, which was already running ESXi 5.0 U1.  This is when the problems began.  At seemingly random intervals, we'd receive the PSOD shown at the top of this post.  Time for an SR to VMware.

In short, new power management features (in the form of CPU throttling as I understand it) were added with ESXi 5 and VMware attempts to utilize them by default.  In our case, we have our BIOS set to a mode that is supposed to not allow the OS (ESXi) any control over the CPU settings, but it does not seem to matter in this case.  Per VMware, even in that scenario, SMIs (System Management Interupts) are still generated and the CPUs are not responding to them in a timely fashion (hence the "didn't have a heartbeat" messages in the PSOD message) and that's when VMware triggers the PSOD.

So at this point I felt like VMware was pointing the finger at HP because they said their internal PR only listed HP DL580 G7s as the affected servers.  Later on it seemed HP was pointing the finger back at VMware.  I, of course, had a support case open with both vendors and at one point HP released a new BIOS update that was rumored to address the problem.  The release notes read:

Problems Fixed:
Addressed a processor issue with Intel Xeon 7500-series Processors and Intel Xeon E7-series Processors that may result in unpredictable system behavior including application level errors, system hangs, Windows blue screens, Linux kernel panics, or a VMware ESX Purple Screen of Death (PSoD).  This issue is not unique to HP ProLiant servers and could impact any system using affected processors.  This revision of the System ROM contains an updated version of Intel's microcode for affected processors that addresses this issue.  The fix for this issue does not impact performance.  Due to the potential severity of the issue addressed in this revision of the System ROM, this System ROM upgrade is considered a critical fix.  HP strongly recommends an immediate update to firmware revisions with required critical fixes.

At this point I was very excited thinking that someone is actually addressing the issue.  Unfortunately, I received a PSOD on one of the servers in less than 24 hours after applying this BIOS update.  Maybe the issue with the Intel CPUs the release notes speak of was something entirely different.

From the beginning, both VMware and HP had a workaround I could apply, but as our servers were not in production yet, I wanted to spend the extra time troubleshooting the issue in hopes for a proper, long-term fix.  But, fast-forward to the end, VMware eventually started getting reports of other vendors' servers being affected--also with the Intel Xeon E7 CPU.

So the bottom line is, VMware has decided that they are going to just disable the power management features that are causing this issue as of ESXi 5.0 U2 when it is released.  Supposedly the code is being re-written for ESXi 6.0 and they are confident the issue will be resolved.  In addition, I was told they aren't even completely sure it will only affect the aforementioned Intel Xeon E7 / 75xx CPUs, so we've made the decision to disable the power management feature on all of our servers.

The setting to disable is an advanced VMkernel boot setting that can be found under the "Software > Advanced Settings > VMkernel > Boot" section and is named "VMkernel.Boot.usePCC" (PCC apparently stands for Power Collaborative Control).  Simply uncheck that option if using the vSphere Client to make the change.  VMware claims this requires a reboot to take effect, though I've seen no indication that this is the case.  Of course, vNugglets.com uses PowerCLI to make the change at build time with the following one-liner:

Set-VMHostAdvancedConfiguration -VMHost <ESXi host name> -Name "VMkernel.Boot.usePCC" -Value $false

However you prefer to do it, if you use Host Profiles, you'll need to keep one more thing in mind.  By default the Host Profile setting "Power system > CPU Policy" is set to "Balanced" if created from a default build of ESXi 5, but if you were to make the above change and then you created the Host Profile from the host, it would be set to "User must explicily choose the policy option" (and yes, that's a VMware typo, not mine).  If you are going back and retroactively changing all your hosts to disable PCC you may find your hosts are out of compliance with the attached Host Profile as it is expecting whatever power management policy you had previously specified and that feature is essentially disabled now.  As a matter of fact, if you tried to apply the Host Profile with a "Power system > CPU Policy" set against a host with PCC disabled, you'd see an error message like so:


To get around this, you'll need to either update the Host Profile from the modified reference host, or simply edit the Host Profile and change the "Power system > CPU Policy" value to "User must explicily choose the policy option" as hinted at earlier.  Then you should be able to re-apply the Host Profile to the affected hosts and it'll disable the option for you, or you can just script it with PowerCLI and verify the hosts are now compliant with your freshly updated Host Profile.

Hopefully this blog post helps someone else that may run into this particular issue with the Intel E7 CPU on ESXi 5.0 U1.

05 August 2012

Indy VMUG Demo Day 2012 PowerCLI Lab

We, as VMUG members in Indianapolis, IN, volunteered to create a PowerCLI lab for this year's Demo Day event.  This post is not to talk about the number of hours we put into it, or how awesome it was (that goes without saying), but is to talk just a bit about the lab guide, make it available, and to share a bit of the on-the-spot code we wrote to manage the lab environment.

The goal of this lab was to continue on from last year's lab, the PowerCLI lab from Demo Day 2011.

This year's lab starts out with some 101/102-level types of exercises and info, and then progresses into 200-level PowerCLI (and maybe a bit beyond).  I cut off the lab guide at thirty (30) pages -- there is so much to cover, and thirty pages got us through about 75% of the material we outlined to discuss.  Now we have leftovers for next time.

This year's lab guide is available at:

http://static.vnugglets.com/vmugdd2012/

and, if you want it for your offline reading pleasure, it is available for download at:

http://static.vnugglets.com/files/VMUGDemoDays2012-PowerCLILabGuide.zip

Have a look.  There are some lab-environment-specific items in the guide, like vCenter server names, VM Host names, VM names, etc., but the knowledge still carries/shines through.  If there are things that do not make sense or on which you have questions, hit us up -- let us know.

The environment the day of the event:  there was definite room for improvement.  The infrastructure was not put in until the day before the event, which makes building it out a bit of a rush, and does not provide for much capacity/performance testing.  Unfortunately, the user experience suffered due to sub-optimal infrastructure performance.

As for managing the lab -- setting up vApps, configuring the View desktops from which users accessed their individual labs, resetting lab environments when users were done with their lab, etc., we had a running doc in which we put code snippets that we used throughout the day.  I have posted it below.

Keep in mind, this is not a polished set of code, not a top-shelf stack of command, not a suggestion on how to run your show -- it is what evolved the day of the event.  We are posting it here for the entertainment/educational value, to show the answer to the "how did you do that on the day of the event?".  So, for fun, here is the mostly-raw code we came up with and used.  Enjoy:
## various code snipplets used in course of managing lab env.
## vNugglets, Jul 2012

## get the vC admin creds
$credVCAdmin = Get-Credential administrator

Connect-VIServer vmug-vc01 -Credential $credVCAdmin


## to start SSH on VMHosts
Get-VMHost | Get-VMHostService | ?{$_.Key -eq "TSM-SSH"} | ?{$_.Running -eq $false} | Start-VMHostService -Confirm:$false
## to stop SSH on VMHosts
Get-VMHost | Get-VMHostService | ?{$_.Key -eq "TSM-SSH"} | ?{$_.Running -eq $true} | Stop-VMHostService -Confirm:$false


## get creds for ESXi hosts
$credESXiRoot = Get-Credential root

## arr of all VMHosts names
$arrVMHostNames = Get-VMHost | %{$_.Name}

## cmd to add the given line to the config file on VMHosts
$strCmdToRun = "echo 'vhv.allow = ```"TRUE```"' >> /etc/vmware/config"
## cmd to echo out the config file
#$strCmdToRun = "cat /etc/vmware/config"
$arrVMHostNames | %{plink.exe -l root -pw rootPasswd $_ $strCmdToRun}



## added add'l portgroup to the std vSwitch0
Get-Cluster | Get-VMHost | Get-VirtualSwitch -Name vSwitch0 | %{New-VirtualPortGroup -VirtualSwitch $_ -Name PowerCLI-32 -VLanId 132 -Confirm:$false}
Get-Cluster | Get-VMHost | Get-VirtualSwitch -Name vSwitch0 | %{New-VirtualPortGroup -VirtualSwitch $_ -Name SRM-32 -VLanId 232 -Confirm:$false}


###############
## clone vApps
## name of cluster in which to make vApps
$strVDIAndMgmtClustername = "VDI and Management"
## prefix name of the vApp
#$strVAppNamePrefix = "SRM"
$strVAppNamePrefix = "PowerCLI"
## the name of the source vApp to clone
$strSourceVAppName = "base_PowerCLI_done"
#$strSourceVAppName = "base_SRM-00_CLONE"

## clone the vApp
21..25 | %{
    $strNewVAppName = "$strVAppNamePrefix-{0:d2}" -f $_
    #$dstToUse = Get-Datastore vapp-vmfs* | sort FreeSpaceMB -Descending:$true | select -First 1
    $dstToUse = Get-Datastore vapp-vmfs* | Get-Random
    New-VApp -Name $strNewVAppName -Location (Get-Cluster $strVDIAndMgmtClustername | Get-VMHost | Get-Random) -Datastore $dstToUse -VApp $strSourceVAppName -RunAsync
} ## end foreach-object

## when clones are done, can move the vApps to the given folder
$strFolderToWhichToMoveVApp = "${strVAppNamePrefix}_vApps"    ## vApp inventory folder is PowerCLI_vApps or SRM_vApps
$oDestFolder = Get-Folder $strFolderToWhichToMoveVApp
Get-VApp "$strVAppNamePrefix*" | ?{$_.ExtensionData.ParentFolder.ToString() -ne $oDestFolder.Id} | Move-VApp -Destination $oDestFolder

## set proper NetworkName for NIC on VMs in the vApps (portgroup names match the vApp names); -NetworkName param is cASeSensitIVe!
21..25 | %{
    $intLabNumber = $_
    $strVAppName = "$strVAppNamePrefix-{0:d2}" -f $intLabNumber
    Get-VApp $strVAppName | Get-VM | Get-NetworkAdapter | Set-NetworkAdapter -NetworkName $strVAppName -Confirm:$false 
} ## end foreach-object

## end clone vApps
###############



###############
## prep vApps
## take snapshots of vApp VMs -- do in PoweredOn or PoweredOff state?
#$strVAppNamePrefix = "SRM"
$strVAppNamePrefix = "PowerCLI"
21..25 | %{
    ## make vApp name
    $strNewVAppName = "$strVAppNamePrefix-{0:d2}" -f $_
    Get-VApp $strNewVAppName | Get-VM | New-Snapshot -Name "0" -Description "Initial state" -Memory -Quiesce -Confirm:$false -RunAsync
} ## end foreach-object

21..25 | %{$strNewVAppName = "$strVAppNamePrefix-{0:d2}" -f $_; Get-VApp $strNewVAppName | Start-VApp -Confirm:$false -RunAsync}
## for the PowerCLI stuff -- restart the ESXi VMs so that vC shows them as powered on
#21..25 | %{$strNewVAppName = "$strVAppNamePrefix-{0:d2}" -f $_; Get-VApp $strNewVAppName | Get-VM esxi* | Restart-VM -Confirm:$false}
21..22 | %{
    ## make vApp name
    $strNewVAppName = "$strVAppNamePrefix-{0:d2}" -f $_
    Get-VApp $strNewVAppName | Get-VM | New-Snapshot -Name "1" -Description "Powered on" -Memory -Quiesce -Confirm:$false -RunAsync
} ## end foreach-object
## end prep vApps
###############


$strVDIVMsBaseName = "PowerCLI"
21..25 | %{Get-VM $("vm-${strVDIVMsBaseName}-{0:d2}" -f $_)} | %{
#Get-VM "vm-${strVDIVMsBaseName}-*" | %{
    $strVMName = $_.Name
    Get-NetworkAdapter -VM $_ -Name "Network Adapter 2" | Set-NetworkAdapter -NetworkName $strVMName.Trim("vm-") -Confirm:$false -WhatIf
} ## end foreach-Object
#-- CHECK ON POWERCLI-06, SRM-03, SRM07 -- not sure 2nd nic is on right virtual network


###############
## reset vApp for next user -- revert all VMs in the vApp to snapshot
$strVAppToReset = "PowerCLI-03"
$strSnapshotName = "1"

$vappToReset = Get-VApp $strVAppToReset
## stop VMs in vApp
#$vappToReset | Get-VM | Stop-VM -Confirm:$false
$vappToReset | Get-VM | Set-VM -Snapshot $strSnapshotName -Confirm:$false

## start the vApp if reverting to snapshot 0 (if snapshot 1, the VMs were on)
#$vappToReset | Start-VApp -Confirm:$false
Get-VM "vm-$strVAppToReset" | Restart-VMGuest -Confirm:$false
## end reset vApp -- revert all VMs in the vApp to snapshot
###############


## reporting stuff
## show snapshots for each vApp
11..20 | %{Get-VApp powercli-$_} | %{$strVAppName = $_.Name; Get-VM -Location $_ | Get-Snapshot | Select @{n="vapp";e={$strVAppName}},Name,VM}

And, other guys besides Allen and Matt of vNugglets.com and the VMUG team that put in work to make the labs happen:  big Joe Filippello (@joefilippello, filippello.com) and Jaron Hilger (@hilgertech).  Thanks, guys.