PowerCLI on steroids – Custom attributes

PowerCLI is a great tool to manage your vSphere environment. No discussion about that!

The package contains cmdlets for, let’s say 80% of your vSphere day to day tasks, and for the missing 20% you can always fall back on the vSphere APIs (or ask in the PowerCLI community).

The only problem that pops up from time to time is the performance of some of the PowerCLI cmdlets.The Get-Vm cmdlet is one of these infamous cmdlets which, when executed in a decently sized (number of guests) vSphere environment, tends to be quite slow.

But as with the 80%-20% rule above, you can fall back on the vSphere APIs to speed things up a bit.
Add to that some medium to advanced PowerShell features and the time gains you can reach are impressive.

The problem at hand was that we needed a script to update a number of custom attributes on our guests.

A quick method of doing this would be something along these lines:

Get-VM | Set-CustomField -Name “MyField” -Value “new-value”

Quick and easy, but you’re in for a surprise when you run this in your 1000+ guest environment. You could have considerably more than 1 coffee while this is running.

The not-so-spectacular speed of the Get-VM cmdlet is common knowledge in the PowerCLI community and in fact confirmed by VMware in several PowerCLI community threads. See for example here and here.

A faster alternative is to use the Get-View cmdlet with the -ViewType VirtualMachine parameter. This cmdlet gets the objects that represent guests directly from the vSphere environment without converting them to PowerCLI objects.

The only problem we have now is that the Get-View cmdlet returns a VirtualMachine (vSPhere) object instead of a VirtualMachineImpl (PowerCLI) object, like the Get-Vm cmdlet does.

Luckily there is a PowerCLI cmdlet that allows us to go from a VirtualMachine object to a VirtualMachineImpl object; the Get-VIObjByVIView cmdlet.

Our script becomes

Get-View -ViewType VirtualMachine | Get-VIObjectByVIView | Set-CustomField -Name "MyField" -Value "new-value"

This will run a bit faster, but not that fast that it would bring down your coffee addiction.

As always, when at a dead-end with PowerCLI look in the APIs.

We find a method called setCustomValue on the ExtensibleManagedObject object.The ExtensibleManagedObject object is extended by the ManagedEntity object and that in it's turn is extended by the VirtualMachine object.

image

And that's the kind of object we had from our Get-View -ViewType VirtualMachine cmdlet. Looks promising.

But wait a minute, the setCustomValue method requires 2 parameters, key and value.The value parameter is easy, that is a string that contains the new content of the custom attribute.But how do we get at the key parameter ?

setCustomValue

For most vSphere objects there is a manager available that provides additional methods and properties for the specific types of objects. The same goes for the custom fields.

From the ServiceInstance we can get at the customFieldsManager. This manager contains one property, called field, which is an array of CustomFieldDef objects.

image

Each CustomFieldDef object defines the name and key of a custom attribute. So this looks the place where we find the link between the custom attribute name and the key.

customfielddef

    Let's store this information in a kind of dictionary (in PS lingo called a hash table).

    We construct two hash tables

  • one where we can look up a key when we know the name of the custom attribute
  • one where we can find the name when we know the key of the custom attribute
  • # Create Custom Field & Name lookup table
    $customFieldMgr = Get-View (Get-View ServiceInstance).Content.CustomFieldsManager
    $customKeyLookup = @{}
    $customNameLookup = @{}
    $customFieldMgr.Field | where {$_.ManagedObjectType -eq "VirtualMachine"} | % {
    $customKeyLookup.Add($_.Key, $_.Name)
    $customNameLookup.Add($_.Name, $_.Key)
    }

Now we need to get our VirtualMachine objects. The fastest way of getting these is with the Get-View -ViewType VirtualMachine cmdlet.

Now we need to consider how we are going to store these objects. For that we better first list what we actually want to do with these objects. In our case we came up with the following requirements:

  • store the VirtualMachine object to be able to use the setCustomValue method
  • store the actual contents of the custom attributes for each guest
  • create a report that lists all guests and their custom attributes
  • import/export the custom attributes from/to a CSV file
  • find all guests that have a specific value in a custom attribute
  • update a specific custom attribute for specific guests

After some trial and error I came up with this structure

image

The outer structure is a hash list with the guest’s name as the key.

The value part of this outer structure is a custom object with two properties:

  • Custom: is another hash table which contains all custom attributes for that guest
  • Object: is the actual VirtualMachine object
  • Based on this structure we can now script all the requirements.

    1) Fill the hash table with all the guests and all their custom attributes for which there is a value.
# Get all guests as VirtualMachine objects
$vms = Get-View -ViewType VirtualMachine
# Create dictionary with all guests and their custom fields
$vmArray = @{}
$vms | % {
$VmCustom = @{}
$_.CustomValue | % {
$VmCustom.Add($_.Key, $_.Value)
}
$row = New-Object psobject
$row | Add-Member -MemberType noteproperty -Name Object -Value $_
$row | Add-Member -MemberType noteproperty -Name Custom -Value $VmCustom
$vmArray.Add($_.Name, $row)
}
    2) Report all guests and their custom fields
# Report all guests and their custom fields
Write-Host "Report all guest with custom fields" -fore red
$vmArray.GetEnumerator() | %{
Write-Host $_.Name
$_.Value.Custom.getenumerator() | %{
Write-Host "t” $customKeyLookup[$_.Name] $_.Value
}
}
    3) Export all custom attributes to a CSV file
# Export all custom attributes to a CSV file (backup)
$list = @()
$vmArray.GetEnumerator() | %{
$vmName = $_.Name
$_.Value.Custom.getenumerator() | %{
$row = New-Object psobject
$row | Add-Member -memberType noteproperty -name VmName -value $vmName
$row | Add-Member -memberType noteproperty -name CAName -value $customKeyLookup[$_.Name]
$row | Add-Member -memberType noteproperty -name CAValue -value $_.Value
$list += $row
}
}
$list | Export-Csv “C:\CA-backup.csv” -noTypeInformation
    4) Import all custom attributes from a CSV file
# Import all custom attributes from a CSV file (restore)
Import-Csv “C:\CA-backup.csv” | %{
$vmArray[$_.VmName].Object.setCustomValue($_.CAName,$_.CAValue)
}
    5) Add/update a custom field
# Add/update a custom field, named MyField# with a new value for all guests whose name starts with “PC”
$field = “MyField”
$newValue = “1234”
if (-not $customNameLookup.ContainsKey($field)){
$customFieldMgr.AddCustomFieldDef($field, “VirtualMachine”, $null, $null)
}
$vmArray.GetEnumerator() | where {$_.Name -like “PC*”} | % {
$_.Value.object.setCustomValue(“MyField”,$newValue)
}
    6) Find all guests that have a specific custom attribute set to a specific value
# Find all guests that have MyField=”yes” (case sensitive)
Write-Host “Find all with MyField=yes” -fore red
$tgtKey = $customNameLookup[“MyField”]
$vmArray.GetEnumerator() | where {$_.Value.Custom[$tgtKey] -ceq “yes”} | %{
$_.Name
}
    7) Update a specific custom field
# Update custom field, named VCBDRM. Change all “yes” to “Yes”
Write-Host “Update all VCBDRM=Yes to VCBDRM=yes” -fore red
$tgtKey = $customNameLookup[“VCBDRM”]
$vmArray.GetEnumerator() | where {$_.Value.Custom[$tgtKey] -ceq “Yes”} | %{
$_.Value.object.setCustomValue(“VCBDRM”,“yes”)
}

Now that we have all the functionality we need it’s time to see if we were able to lower the execution time.In my limited test environment I was able to execute the custom attribute updates in 1/40th of the original time.

For a test in a bigger environment I got the help from Arnim van Lieshout (thanks Arnim).

He was able to bring the execution time of one custom attribute update on all the guests in his VI environment down from 70 seconds to less than 1 second.

Arnim’s environment contained 842 guests.

It would mean that he could do his planned updates of the custom attributes on all the guests in less than 1 hour, where he previously would have needed more than 65 hours.

In the mean time I received confirmation from Carter Shanklin that there is indeed a bug in the current release of the Set-CustomField cmdlet that could explain part of this behaviour.

My conclusion, it is sometimes useful to dive into the vSphere APIs and to learn the more advanced PowerShell techniques, like hash tables.

16 thoughts on “PowerCLI on steroids – Custom attributes”

  1. Hi,

    I am trying to deploy vAPP in specific cluster.

    My problem is, I will be knowing the cluster name where i can deploy the vAPP but as part of My .ovf description i need to select data store in the cluster where i am deploying.
    i need to select max free size datastore.

    please help on this;

    PowerCLI C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI> $ds=Get-Data
    store -VMHost xxxxxxxxxx | select FreeSpaceGB | where

    wt shouold be criteria i should specify??

    vishwa

  2. Hi All,

    i am very new to Powercli, i have a specific requirement of automating the v app deployment. where vapp will have 5 vms and i need to set ip/user agreement/and other customised properties… can any one help me,,, i am in great trouble. plz help me

  3. I need help reading in a custom attribute into my healthcheck script. Here is the section that returns the vm information. I want to retreive a custom attribute called ServerOwner. I am able to read the information using Get-Annotation -name “ServerOwner” but cannot figure out how to inject that in the current powercli script. Any help would be greatly appreciated.

    ##################
    # VM information #
    ##################
    $Report = @()

    get-vm | % {
    $vm = Get-View $_.ID
    $vms = “” | Select-Object VMName, Hostname, IPAddress, ToolsStatus, TotalCPU, TotalMemory, MemoryUsage, VMState
    $vms.VMName = $vm.Name
    $vms.HostName = $vm.guest.hostname
    $vms.IPAddress = $vm.guest.ipAddress
    $vms.VMState = $vm.summary.runtime.powerState
    $vms.TotalCPU = $vm.summary.config.numcpu
    $vms.TotalMemory = $vm.summary.config.memorysizemb
    $vms.MemoryUsage = $vm.summary.quickStats.guestMemoryUsage
    $vms.ToolsStatus = $vm.guest.toolsstatus

    $Report += $vms
    }
    $Report | ConvertTo-Html –title “Virtual Machine information” –body “Virtual Machine information.” -head “” | Out-File -Append $filelocation

  4. Hi,

    The script is a great start for me, please could you help me with the following.

    I have a custom annotation called veeam this gets populated with the last veeam backup time.

    What i would like to do is produce a list of vms that do not have a value or have not been backed up in x days.

    any help would be fantastic.

    regards,

  5. Vladimir, this post, which I did as a guest blogger on Alan’s blog btw, is quite old.
    No, the current cmdlets are much improved regarding execution speed.
    You still might notice a better speed when you the SDK methods, but the difference will not be that dramatic anymore.

  6. Hi Alan,

    Thank you for scripts and explanations.
    Just one question. Do current version have the same performance problems on some cmdlets as it was with previous ones?
    Do I have to use the same technique you mentioned in this topic in nowadays?

    Thank you

  7. Hi Alan,

    Many thanks for putting in the time to work this out. It is exactly what I was looking for. I have tested this and it works for me, however I have a scenario where custom fields were entered as “Global” instead of “Virtual Machine”. In this instance the fields returned are only VmName and CAValue. CAName is blank. I don’t have the PowerCLI/Powershell skills to work out what is wrong. Any ideas..?

    Cheers

    Glenn

  8. Hello Alan,

    again thank you for your reply:

    i think there is still something wrong, i just added the lines but get the same error:

    [vSphere PowerCLI] C:\Scripts> .\export_cf.ps1

    Name Port User
    —- —- —-
    vcserverold 443 vcadmin
    You cannot call a method on a null-valued expression.
    At C:\Scripts\export_cf.ps1:5 char:23
    + $vmArray.GetEnumerator( <<<< ) | %{

  9. Hello Alan,

    i need to move my vcenter (or all esx hosts) to my new vsphere vcenter. I just connect 3.5 esx hosts to my new vcenter.
    regarding to this nice guide iam able to move custom fields from old vcenter to my vsphere vcenter using export-csv.

    http://www.virtu-al.net/2009/05/29/powercli-on-steroids-custom-attributes/

    —–snip———snip———snip———snip—-

    1.Export all custom attributes to a CSV file (backup)
    $list = @()
    $vmArray.GetEnumerator() | %{
    $vmName = $_.Name
    $_.Value.Custom.getenumerator() | %{
    $row = New-Object psobject
    $row | Add-Member -memberType noteproperty -name VmName -value $vmName
    $row | Add-Member -memberType noteproperty -name CAName -value $customKeyLookuphttp://$_.Name
    $row | Add-Member -memberType noteproperty -name CAValue -value $_.Value
    $list += $row
    }
    }
    $list | Export-Csv “C:\CA-backup.csv” -noTypeInformation
    —–snip———snip———snip———snip———snip—-

    so i connect vi PowerCli to my old vcenter “connect-viserver” and export all CF, “reconnect” all 3.5 esx server to my vsphere vcenter, connect to my new vcenter and import them accordingly:

    —–snip———snip———snip———snip———snip—-
    # Import all custom attributes from a CSV file (restore)
    Import-Csv “C:\CA-backup.csv” | %{
    $vmArrayhttp://$_.VmName.Object.setCustomValue($_.CAName,$_.CAValue)
    }
    —–snip———snip———snip———snip———snip—-

    I can’t run this code snippets as a script itself and my PS Skills are limited so here is my Question: Could someone help me to get started ?
    I guess i need to connect to my vcenter first.. like “Connect-VIServer vcserver”

    your help is much appreciated. thank you
    rgrds uxmax

    EDIT: so i just add the connection lines in the appropriate (export/import) code “snippets” ?

    Great Site btw 🙂

Leave a Reply