Identifying and fixing VMs Affected By SvMotion / VDS Issue

April 17, 2012 Alan PowerShell, VMware, 27

Duncan Epping recently described an issue with virtual machines (VMs) which have moved via Storage vMotion (SvMotion) and are connected to a vNetwork Distributed Switch (VDS), if you are using a configuration where VMs are connected to a VDS and could potentially move via SvMotion then please make sure you read his article here.

William recently showed how we could check for this issue using Perl, on this post you will see a similar script which uses PowerCLI to look for the issue and also resolve the issue fixing the VMs which could potentially have an issue.

In this script I use the VMware VDS Fling which adds VDS cmdlets to PowerCLI, more information and lots of examples on this fling can be found here.  Please make sure you have it installed before using this script and are using a 32 bit PowerShell or PowerCLI console.

Using the script

To check the VMs we can easily pipe a list of VMs into our function which can be seen below.  This can be all VMs in a Cluster, all VMs on a particular host or any other list of VMs you can think of, for my examples below I have shown all VMs attached to a vCenter

As you can see from the above screenshot, all VMs are fine apart from VM12 which currently has the problem described in Duncan’s article,  now to fix the issue.

We can use the same script with a –Fix parameter which allows us to fix the issue, when fixing the issue the script will move each of the VMs network connections to a new port on the same portgroup and then move it back again to its original port.   If no free ports are available the script will expand your portgroup temporarily and then decrease the ports when finished.

As you can see from the above screenshot, the issue has now been resolved for this VM by using the function with the –Fix parameter and further running of the script in test mode will show all are now fine.

UPDATE: The script has now been updated to support remediation for VMs connected to both a VMware VDS as well as Cisco N1KV. The solution, thanks to one of our internal engineers was to “move” the VM’s dvport from one to another, all while staying within the existing dvPortgroup which will also force the creation of the .dvsdb port file. Once the dvport move has successfully completed, we will move it back to it’s original dvport that it initially resided on. We no longer have to rely on creating a temporally dvPortgroup and best of all, we can now remediate both VDS and N1KV.

Disclaimer: This script is not officially supported by VMware, please test this in a development environment before using on production systems.

The Script

If (-Not (Get-PSSnapin VMware.VimAutomation.VdsComponent -WarningAction SilentlyContinue) ) {
	Add-PSSnapin VMware.VimAutomation.VdsComponent
}

Function Test-VDSVMIssue {
	Param (
		[parameter(Mandatory=$true,ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
        [PSObject[]]$VM,
		[switch]$Fix
	)
	Process {
		Foreach ($VMachine in $VM){
			Foreach ($NA in ($VMachine | Get-NetworkAdapter)) {
				$VMName = $VMachine.Name
				If (($NA.ExtensionData.Backing.GetType()).Name -eq "VirtualEthernetCardDistributedVirtualPortBackingInfo") {
					$PortKey = $NA.ExtensionData.Backing.Port.PortKey
					$vSwitchID = $NA.ExtensionData.Backing.Port.SwitchUUID
					$Datastore = (($VMachine.ExtensionData.Config.Files.VmPathName).split("]")[0]).Replace("[","")
					$filename = "$($datastore):\.dvsData\$vSwitchID\$PortKey"
					if (-not (Get-PSDrive $datastore -ErrorAction SilentlyContinue)) {
						$NewDrive = New-PSDrive -Name $Datastore -Location (Get-Datastore $Datastore) -PSProvider VimDatastore -Root '\'
					}
					$filecheck = Get-ChildItem -Path $filename -ErrorAction SilentlyContinue
					if ($filecheck) {
						Write-Host -ForegroundColor Green "$VMName $($NA.Name) is OK"
					} Else {
						Write-Host -ForegroundColor Red "Problem found with $VMName $($NA.Name)"
						If ($Fix) {
							Write-Host -ForegroundColor Yellow "Fixing issue..."
							$VDSPG = Get-VdsDistributedPortgroup $NA.NetworkName
							$DVPort = $null
							Write-Host -ForegroundColor Yellow "..Finding free port on $($NA.NetworkName)"
							$DVPort = Get-VdsDVPort -DVPortgroup $VDSPG -Active:$false | Select -last 1
							$Move = $True
							if (-not $DVPort) {
								Write-Host -ForegroundColor Yellow "..No free ports found on $($VDSPG.Name), adding an additional port"
								If (($VDSPG.PortBinding -eq "Ephemeral") -or ($VDSPG.PortBinding -eq "Dynamic")) {
									Write "Unable to add a port to $($NA.NetworkName) since dvportgroup is configured as $($VDSPG.PortBinding)"
									Write-Host -ForegroundColor Red "Problem still exists with $VMName please resolve manually"
									$Move = $false
								} Else {
									$CurrentPorts = $VDSPG.NumPorts 
									$NewTotalPorts = $VDSPG.NumPorts + 1
									Set-VdsDistributedPortgroup -NumPorts $NewTotalPorts -DVPortgroup $VDSPG | Out-Null
									$PGAdded = $true
									$DVPort = Get-VdsDVPort -DVPortgroup $VDSPG -Active:$false | Select -last 1
								}
							}
							If ($Move){
								Write-Host -ForegroundColor Yellow "..Moving $($NA.Name) to another free port on $($VDSPG.Name)"
								$NA | Set-NetworkAdapter -PortKey $DVPort.Key -DistributedSwitch $VDSPG.VirtualSwitch -Confirm:$false | Out-Null
								Write-Host -ForegroundColor Yellow "..Moving $($NA.Name) back to port $PortKey"
								$NA | Set-NetworkAdapter -PortKey $PortKey -DistributedSwitch $VDSPG.VirtualSwitch -Confirm:$false | Out-Null
								Write-Host -ForegroundColor Yellow "..Checking changes were completed"
								$filecheck = Get-ChildItem -Path $filename -ErrorAction SilentlyContinue
								if ($filecheck) {
									Write-Host -ForegroundColor Green "$VMName $($NA.Name) is now fixed and OK"
								} Else {
									Write-Host -ForegroundColor Red "Problem still exists with $VMName please resolve manually"
								}
								If ($PGAdded) {
									Write-Host -ForegroundColor Yellow "..Removing the added port on $($VDSPG.Name)"
									Set-VdsDistributedPortgroup -NumPorts $CurrentPorts -DVPortgroup $VDSPG | Out-Null
									$PGAdded = $false
								}
							}
						}
					}	
				} Else {
					Write-Host -ForegroundColor Green "$VMName is not connected to a dvSwitch so this issue is not relevant."
				}
			}
		}
		Get-PSDrive | Where { ($_.Provider -like "*VimDatastore") -and ( $_.Name -notlike "*vmstore*")} | Foreach {
			Remove-PSDrive $_ | Out-Null
		}
	}
}

-Alan

PowerCLI, PowerShell, VMware

Silicon Valley VMUG – April 11th Carolina VMUG–15th May 2012