As part of a large scale data analysis project I was working on recently I used Horizon View and Instant clones to allow me to deploy hundreds of VMs based on my original templated VM which had my app I wanted to use to take in the data, transform it and then return a result. Its important to note that this app was never written to work in a batch mode, it literally took one input and gave an output based on a number of factors.

Whilst this post is not about the benefits of Instant Clones, let me tell you, once I had the original template VM working correctly and optimized for performance it worked like a charm, there is something to be said for the simplicity and efficiency of instant clones and the memory sharing techniques it uses to be able to run hundreds of essentially the same VM.

Once I had these VMs deployed via Horizon I could easily send a job to each VM and tell it to run the job, as these jobs however took an indeterminate amount of time to crunch the data, I needed a way to monitor them and let me know when all the jobs had finished on all the VMs so I could pull the data to analyze it.

First I went down the route of using Invoke-VMScript to hook inside the VM’s and see if the process was running, this however took a long time to complete on 100’s of virtual machines and the monitoring job often took over 20 minutes to monitor the VMs and tell me if the job was completed… far too long for what I needed. So after some googling I learned from my good friend William Lam that there was a new API available that through VMtools would update the GuestInfo with the processes running inside the VM on a configurable timely basis (see his post here), this allowed me to essentially push the information externally to the VM Guest Operating system and grab the info when I needed it to see if my process was running.

William had also written a handy function which I adjusted to work with an array of VMs and tell me if the process was running.

Using these new found skills I was easily able to write a function that allowed me to pull the running processes from the VMs and remove them from the list as that process finished inside the VM, for the icing on the cake I even got it to update me in slack on how it was doing 😉

So i thought I would share this script and encourage you to think about using this way which is much easier and a more performant way to pull the results of the running processes..

Monitoring Script

Function Send-SlackMessage ($Channel = "#MyProjectChannel", $Message) {
        
    $payload = @{
        "channel"    = $Channel
        "icon_emoji" = ":datacenter:"
        "text"       = $Message
        "username"   = "DC Script"
    }

    Invoke-WebRequest `
        -UseBasicParsing `
        -Body (ConvertTo-Json -Compress -InputObject $payload) `
        -Method Post `
        -Uri "https://myslackhookurl" | Out-Null
}

Function Get-VMApplicationInfo {
<#
    .DESCRIPTION Retrieves discovered applications running inside of a VM
    .NOTES  Author:  William Lam
    .NOTES  Site:    www.virtuallyghetto.com
    .NOTES  Reference: http://www.virtuallyghetto.com/2019/12/application-discovery-in-vsphere-with-vmware-tools-11.html
    .PARAMETER VM
        VM Object
    .PARAMETER Output
        CSV or JSON output file
    .EXAMPLE
        Get-VMApplicationInfo -VM (Get-VM "DC-01")
    .EXAMPLE
        Get-VMApplicationInfo -VM (Get-VM "DC-01") -UniqueOnly
    .EXAMPLE
        Get-VMApplicationInfo -VM (Get-VM "DC-01") -Output CSV
    .EXAMPLE
        Get-VMApplicationInfo -VM (Get-VM "DC-01") -Output JSON
#>
    param(
        [Parameter(Mandatory=$true)]$VM,
        [Parameter(Mandatory=$false)][ValidateSet("CSV","JSON")][String]$Output,
        [Parameter(Mandatory=$false)][Switch]$UniqueOnly
    )

    $appInfoValue = (Get-AdvancedSetting -Entity $VM -Name "guestinfo.appInfo").Value

    if($appInfoValue -eq $null) {
        Write-Host "Application Discovery has not been enabled for this VM"
    } else {
        $appInfo = $appInfoValue | ConvertFrom-Json
        $appUpdateVersion = $appInfo.updateCounter

        if($UniqueOnly) {
            $results = $appInfo.applications | Sort-Object -Property a -Unique| Select-Object @{Name="Application";e={$_.a}},@{Name="Version";e={$_.v}}
        } else {
            $results = $appInfo.applications | Sort-Object -Property a | Select-Object @{Name="Application";e={$_.a}},@{Name="Version";e={$_.v}}
        }

        Write-verbose "Application Discovery Time: $($appInfo.publishTime)"
        if($Output -eq "CSV") {
            $fileOutputName = "$($VM.name)-version-$($appUpdateVersion)-apps.csv"

            Write-Host "`tSaving output to $fileOutputName"
            ($appInfo.applications) | ConvertTo-Csv | Out-File -FilePath "$fileOutputName"
        } elseif ($Output -eq "JSON") {
            $fileOutputName = "$($VM.name)-version-$($appUpdateVersion)-apps.json"

            Write-Host "`tSaving output to $fileOutputName"
            ($appInfo.applications) | ConvertTo-Json | Out-File -FilePath "$fileOutputName"
        } else {
            $results
        }
    }
}


Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -ErrorAction SilentlyContinue -Confirm:$false -DisplayDeprecationWarnings $false | out-null
$ProgressPreference = 'SilentlyContinue'


Connect-viserver vcsa-01a.myenv.local -username Administrator@vsphere.local -password VMware1! | Out-Null
$vms = get-vm InstantCloneVM* | Where-Object {$_.PowerState -eq "PoweredOn"} | Sort-Object Name
Send-Slackmessage -message "Monitor Job Started" 
do {
    foreach ($vm in $vms) {
        Write-Host "Checking $VM status..."
        $result = Get-VMApplicationInfo -VM $VM| Where { $_.Application -eq "MyAppProcess.exe" }
        if (!$result){
            Write-Host "Removing $vm as it has completed"
            $vms = $vms | where { $_.Name -ne $vm.name}
        } else {
            Write-Host "$vm still running"
        }
        if (($vms.count -lt 10) -and (! $sentmail)){
            Send-Slackmessage -message "Less than 10 Sigma VMs left"
            $sentmail = $true
        }
    }
    if (! ($oldnumvms -eq ($vms.count))){
        Send-Slackmessage -message "$($vms.count) VMs still running"
    }
    start-sleep -s 30
    $oldnumvms = $vms.count
}
while ($vms)
Send-Slackmessage -message  "All Jobs Completed"