Identifying and fixing VMs Affected By SvMotion / VDS Issue

Duncan Epping recently described an issue with virtual machines (VMs) which have moved via Storage vMotion (SvMotion) and are connected to a vNetwork Distributed Switch (VDS), if you are using a configuration where VMs are connected to a VDS and could potentially move via SvMotion then please make sure you read his article here.

William recently showed how we could check for this issue using Perl, on this post you will see a similar script which uses PowerCLI to look for the issue and also resolve the issue fixing the VMs which could potentially have an issue.

In this script I use the VMware VDS Fling which adds VDS cmdlets to PowerCLI, more information and lots of examples on this fling can be found here.  Please make sure you have it installed before using this script and are using a 32 bit PowerShell or PowerCLI console.

Using the script

To check the VMs we can easily pipe a list of VMs into our function which can be seen below.  This can be all VMs in a Cluster, all VMs on a particular host or any other list of VMs you can think of, for my examples below I have shown all VMs attached to a vCenter

image

As you can see from the above screenshot, all VMs are fine apart from VM12 which currently has the problem described in Duncan’s article,  now to fix the issue.

We can use the same script with a –Fix parameter which allows us to fix the issue, when fixing the issue the script will move each of the VMs network connections to a new port on the same portgroup and then move it back again to its original port.   If no free ports are available the script will expand your portgroup temporarily and then decrease the ports when finished.

image

As you can see from the above screenshot, the issue has now been resolved for this VM by using the function with the –Fix parameter and further running of the script in test mode will show all are now fine.

UPDATE: The script has now been updated to support remediation for VMs connected to both a VMware VDS as well as Cisco N1KV. The solution, thanks to one of our internal engineers was to “move” the VM’s dvport from one to another, all while staying within the existing dvPortgroup which will also force the creation of the .dvsdb port file. Once the dvport move has successfully completed, we will move it back to it’s original dvport that it initially resided on. We no longer have to rely on creating a temporally dvPortgroup and best of all, we can now remediate both VDS and N1KV.

Disclaimer: This script is not officially supported by VMware, please test this in a development environment before using on production systems.

The Script

If (-Not (Get-PSSnapin VMware.VimAutomation.VdsComponent -WarningAction SilentlyContinue) ) {
	Add-PSSnapin VMware.VimAutomation.VdsComponent
}

Function Test-VDSVMIssue {
	Param (
		[parameter(Mandatory=$true,ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
        [PSObject[]]$VM,
		[switch]$Fix
	)
	Process {
		Foreach ($VMachine in $VM){
			Foreach ($NA in ($VMachine | Get-NetworkAdapter)) {
				$VMName = $VMachine.Name
				If (($NA.ExtensionData.Backing.GetType()).Name -eq "VirtualEthernetCardDistributedVirtualPortBackingInfo") {
					$PortKey = $NA.ExtensionData.Backing.Port.PortKey
					$vSwitchID = $NA.ExtensionData.Backing.Port.SwitchUUID
					$Datastore = (($VMachine.ExtensionData.Config.Files.VmPathName).split("]")[0]).Replace("[","")
					$filename = "$($datastore):\.dvsData\$vSwitchID\$PortKey"
					if (-not (Get-PSDrive $datastore -ErrorAction SilentlyContinue)) {
						$NewDrive = New-PSDrive -Name $Datastore -Location (Get-Datastore $Datastore) -PSProvider VimDatastore -Root '\'
					}
					$filecheck = Get-ChildItem -Path $filename -ErrorAction SilentlyContinue
					if ($filecheck) {
						Write-Host -ForegroundColor Green "$VMName $($NA.Name) is OK"
					} Else {
						Write-Host -ForegroundColor Red "Problem found with $VMName $($NA.Name)"
						If ($Fix) {
							Write-Host -ForegroundColor Yellow "Fixing issue..."
							$VDSPG = Get-VdsDistributedPortgroup $NA.NetworkName
							$DVPort = $null
							Write-Host -ForegroundColor Yellow "..Finding free port on $($NA.NetworkName)"
							$DVPort = Get-VdsDVPort -DVPortgroup $VDSPG -Active:$false | Select -last 1
							$Move = $True
							if (-not $DVPort) {
								Write-Host -ForegroundColor Yellow "..No free ports found on $($VDSPG.Name), adding an additional port"
								If (($VDSPG.PortBinding -eq "Ephemeral") -or ($VDSPG.PortBinding -eq "Dynamic")) {
									Write "Unable to add a port to $($NA.NetworkName) since dvportgroup is configured as $($VDSPG.PortBinding)"
									Write-Host -ForegroundColor Red "Problem still exists with $VMName please resolve manually"
									$Move = $false
								} Else {
									$CurrentPorts = $VDSPG.NumPorts 
									$NewTotalPorts = $VDSPG.NumPorts + 1
									Set-VdsDistributedPortgroup -NumPorts $NewTotalPorts -DVPortgroup $VDSPG | Out-Null
									$PGAdded = $true
									$DVPort = Get-VdsDVPort -DVPortgroup $VDSPG -Active:$false | Select -last 1
								}
							}
							If ($Move){
								Write-Host -ForegroundColor Yellow "..Moving $($NA.Name) to another free port on $($VDSPG.Name)"
								$NA | Set-NetworkAdapter -PortKey $DVPort.Key -DistributedSwitch $VDSPG.VirtualSwitch -Confirm:$false | Out-Null
								Write-Host -ForegroundColor Yellow "..Moving $($NA.Name) back to port $PortKey"
								$NA | Set-NetworkAdapter -PortKey $PortKey -DistributedSwitch $VDSPG.VirtualSwitch -Confirm:$false | Out-Null
								Write-Host -ForegroundColor Yellow "..Checking changes were completed"
								$filecheck = Get-ChildItem -Path $filename -ErrorAction SilentlyContinue
								if ($filecheck) {
									Write-Host -ForegroundColor Green "$VMName $($NA.Name) is now fixed and OK"
								} Else {
									Write-Host -ForegroundColor Red "Problem still exists with $VMName please resolve manually"
								}
								If ($PGAdded) {
									Write-Host -ForegroundColor Yellow "..Removing the added port on $($VDSPG.Name)"
									Set-VdsDistributedPortgroup -NumPorts $CurrentPorts -DVPortgroup $VDSPG | Out-Null
									$PGAdded = $false
								}
							}
						}
					}	
				} Else {
					Write-Host -ForegroundColor Green "$VMName is not connected to a dvSwitch so this issue is not relevant."
				}
			}
		}
		Get-PSDrive | Where { ($_.Provider -like "*VimDatastore") -and ( $_.Name -notlike "*vmstore*")} | Foreach {
			Remove-PSDrive $_ | Out-Null
		}
	}
}

28 thoughts on “Identifying and fixing VMs Affected By SvMotion / VDS Issue

  1. Pingback: Fix Vds Basic Provider Error 1 Vmware Windows XP, Vista, 7, 8 [Solved]

  2. Knut

    I’m having some trouble with this script. I run the command “Get-VM | Test-VDSVMIssue” and after a few minutes it’s done, but no output… so I’m thinking it didn’t work.

    When I run just the command “Get-VM”, it lists all the VMs.
    When I run just the command “Test-VDSVMIssue” I get this:
    cmdlet Test-VDSVMIssue at command pipeline position 1
    Supply values for the following parameters:
    VM[0]:
    If I try to enter a VM manually at that line I get this error:
    Get-NetworkAdapter : Cannot process argument transformation on parameter ‘VM’.
    Strings as pipeline input are not supported.

    Any ideas?

  3. Pingback: Storage vMotion / dvSwitch patch released (but manual fix required) | Virten.net

  4. Pingback: VMware vCenter Server 5.0 Update 1a released « Aravind Sivaraman

  5. Pingback: July 2012 ESXi and vCenter updates « alpacapowered

  6. Pingback: HA / Distributed vSwitch problems after Storage vMotion scripts available for KB2013639 « blocksandbytes

  7. Pingback: Storage vMotion and dvSwitch / HA problem explained | Virten.net

  8. Pingback: Updated Script to fix VDS/SvMotion issue | | Virtu-Al.NetVirtu-Al.Net

  9. Pingback: 143: Kritik bir HA-dvS problemi « Orcunus.cOm

  10. Pingback: Storage vMotion and dvSwitch / HA problem explained | Virten.net

  11. Mark Strong

    Hi Alan,
    The script correctly identifies affected VMs in my environment but it failes when I add -Fix command.
    ———————–
    Fixing issue…
    Get-VdsDistributedPortgroup : 03/05/2012 14:46:37 Get-VdsDistributedPortgrou
    p Unable to cast object of type ‘VMware.Vim.DVPortSetting’ to type ‘VMwa
    re.Vim.VMwareDVSPortSetting’.
    At line:26 char:65
    + $VDSPG = Get-VdsDistributedPortgroup <<<< $NA.Ne
    tworkName
    + CategoryInfo : NotSpecified: (:) [Get-VdsDistributedPortgroup],
    VimException
    + FullyQualifiedErrorId : Core_BaseCmdlet_UnknownError,VMware.VimAutomatio
    n.VdsComponent.Commands.Cmdlets.GetDVPortgroup
    ———————–

    In our environment all Port Groups have "_" – underscore in the Port Group name. Can it be the problem?

    Thank you.

  12. Michael Friesen

    More weird issues with powershell scripts started occurring (not just with this one!) so i ran teh script on another machine and it worked correctly! Sorry for bothering you before thinking of trying that!

  13. Pingback: Scripts release for Storage vMotion / HA problem « vResource.net

  14. Michael Friesen

    I cannot get the script to work. There is an error with the $VDSPG.VirtualSwitch being null.

    ..Moving Network adapter 1 to another free port on
    Set-NetworkAdapter : Cannot validate argument on parameter ‘DistributedSwitch’. The argument is null. Supply a non-null
    argument and try the command again.
    At C:\Users\friesem1\scripts\Test-VDSVMIssue.ps1:58 char:73
    + $NA | Set-NetworkAdapter -PortKey $DVPort.Key -DistributedSwitch <<<< $VDSPG.VdsId –
    Confirm:$false | Out-Null
    + CategoryInfo : InvalidData: (:) [Set-NetworkAdapter], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,VMware.VimAutomation.ViCore.Cmdlets.Commands.VirtualDev
    ice.SetNetworkAdapter

    I am using the following PowerCLI versions:
    VMware VDS vSphere PowerCLI Component 4.1.1 build 000001
    VMWare vSphere PowerCLI 4.1 U1 build 332441

    Any ideas regarding the problem? I noticed that I cannot get anything but a null value returned when calling $VDSPG.VirtualSwitch, but can get a value when doing $VDSPG | Select -property VirtualSwitch
    Thanks!

  15. akopel

    So one minor issue I had with the script was in our VDI environment. Many of the ‘non-active’ ports were simply shutdown VMs so when doing the Get-vdsdvport function, it would return an inactive port, but a shutdown VM is still connected so it fails. The simple fix is to also add the -Connected:$false to this line in the script:

    $DVPort = Get-VdsDVPort -DVPortgroup $VDSPG -Active:$false -Connected:$false | Select -last 1

  16. Allan Christiansen

    Updated the script so that it can report the vm’s affected to the pipe


    If (-Not (Get-PSSnapin VMware.VimAutomation.VdsComponent -WarningAction SilentlyContinue) ) {
    Add-PSSnapin VMware.VimAutomation.VdsComponent
    }

    Function Test-VDSVMIssue {
    Param (
    [parameter(Mandatory=$true,ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
    [PSObject[]]$VM,
    [switch]$Fix,
    [Switch]$ReportVM
    )
    Process {
    Foreach ($VMachine in $VM){
    $Vmreport = $Null
    Foreach ($NA in ($VMachine | Get-NetworkAdapter)) {
    $VMName = $VMachine.Name
    If (($NA.ExtensionData.Backing.GetType()).Name -eq “VirtualEthernetCardDistributedVirtualPortBackingInfo”) {
    $PortKey = $NA.ExtensionData.Backing.Port.PortKey
    $vSwitchID = $NA.ExtensionData.Backing.Port.SwitchUUID
    $Datastore = (($VMachine.ExtensionData.Config.Files.VmPathName).split(“]”)[0]).Replace(“[“,””)
    $filename = “$($datastore):\.dvsData\$vSwitchID\$PortKey”
    if (-not (Get-PSDrive $datastore -ErrorAction SilentlyContinue)) {
    $NewDrive = New-PSDrive -Name $Datastore -Location (Get-Datastore $Datastore) -PSProvider VimDatastore -Root ‘\’ -erroraction silentlycontinue
    }
    $filecheck = Get-ChildItem -Path $filename -ErrorAction SilentlyContinue
    if ($filecheck) {
    Write-Verbose “$VMName $($NA.Name) is OK”
    } Else {
    Write-Verbose “Problem found with $VMName $($NA.Name)”
    $Vmreport = $VM
    If ($Fix) {
    Write-Verbose “Fixing issue…”
    $VDSPG = Get-VdsDistributedPortgroup $NA.NetworkName
    $DVPort = $null
    Write-Verbose “..Finding free port on $($NA.NetworkName)”
    $DVPort = Get-VdsDVPort -DVPortgroup $VDSPG -Active:$false | Select -last 1
    $Move = $True
    if (-not $DVPort) {
    Write-Verbose “..No free ports found on $($VDSPG.Name), adding an additional port”
    If (($VDSPG.PortBinding -eq “Ephemeral”) -or ($VDSPG.PortBinding -eq “Dynamic”)) {
    Write “Unable to add a port to $($NA.NetworkName) since dvportgroup is configured as $($VDSPG.PortBinding)”
    Write-Verbose “Problem still exists with $VMName please resolve manually”
    $Move = $false
    } Else {
    $CurrentPorts = $VDSPG.NumPorts
    $NewTotalPorts = $VDSPG.NumPorts + 1
    Set-VdsDistributedPortgroup -NumPorts $NewTotalPorts -DVPortgroup $VDSPG | Out-Null
    $PGAdded = $true
    $DVPort = Get-VdsDVPort -DVPortgroup $VDSPG -Active:$false | Select -last 1
    }
    }
    If ($Move){
    Write-Verbose “..Moving $($NA.Name) to another free port on $($VDSPG.Name)”
    $NA | Set-NetworkAdapter -PortKey $DVPort.Key -DistributedSwitch $VDSPG.VirtualSwitch -Confirm:$false | Out-Null
    Write-Verbose “..Moving $($NA.Name) back to port $PortKey”
    $NA | Set-NetworkAdapter -PortKey $PortKey -DistributedSwitch $VDSPG.VirtualSwitch -Confirm:$false | Out-Null
    Write-Verbose “..Checking changes were completed”
    $filecheck = Get-ChildItem -Path $filename -ErrorAction SilentlyContinue
    if ($filecheck) {
    Write-Verbose “$VMName $($NA.Name) is now fixed and OK”
    } Else {
    Write-Verbose “Problem still exists with $VMName please resolve manually”
    }
    If ($PGAdded) {
    Write-Verbose “..Removing the added port on $($VDSPG.Name)”
    Set-VdsDistributedPortgroup -NumPorts $CurrentPorts -DVPortgroup $VDSPG | Out-Null
    $PGAdded = $false
    }
    }
    }
    }
    } Else {
    Write-Verbose “$VMName is not connected to a dvSwitch so this issue is not relevant.”
    }
    }
    }
    Get-PSDrive | Where { ($_.Provider -like “*VimDatastore”) -and ( $_.Name -notlike “*vmstore*”)} | Foreach {
    Remove-PSDrive $_ | Out-Null
    }
    If (($ReportVM) -and ($VMreport -ne $null)) {
    $VMReport
    }
    }
    }

  17. Erik Bussink

    Hiya Alan,

    Thanks so much for your Test-VDSVMIssue function. It works.

    Is there anyway to output to a log file the description of the VM that are affected with the missing dvPort setting ?

    I’m running the script against a Cluster with 600 VMs and I have about 20% of the VMs that are affected due to SDRS being in Automatic mode.

    Thanks,
    Erik

  18. Stefan

    My problem seems to be related to the fact that we have / (forward slash) in the naming convention of distributed portgroups. I’m working on a fix, but the original script is working fine for portgroups without / in their names 🙂

  19. Stefan

    Hi!
    I’ve been testing this script but am unable to get it working.
    The first part, checking which virtual servers are affected is working fine but the second part where I ass -Fix doesn’t work.
    I’m getting the following error:

    New-VdsDistributedPortgroup : Cannot bind parameter 'Vds'. Cannot convert the "
    " value of type "System.Management.Automation.PSCustomObject" to type "VMware.V
    imAutomation.VdsComponent.Types.V1.VdsVSphereDistributedSwitch".
    At C:\SCRIPTS\vmware_skript\vds_svmotion_check.ps1:34 char:56
    + New-VdsDistributedPortgroup -Name $NewName -V
    ds <<<< (Get-vds | Where {$_.Key -eq $NA.ExtensionData.Backing.Port.SwitchUUID
    }) -ReferenceDVPortgroup $VDSPG | Out-Null
    + CategoryInfo : InvalidArgument: (:) [New-VdsDistributedPortgrou
    p], ParameterBindingException
    + FullyQualifiedErrorId : CannotConvertArgumentNoMessage,VMware.VimAutomat
    ion.VdsComponent.Commands.Cmdlets.NewDVPortgroup

    This results is the new dv-portgroup not being created and as such, the rest of the script also throws out errors.

    Any tips?

  20. Alan Post author

    Thanks Alpaca, I updated the post to link to the VDS fling – well spotted. Thanks for the further info too !

  21. Pingback: Problem with vSphere 5 HA and Storage vMotion when VM on vDS « Long White Virtual Clouds

  22. Alpaca

    Jon:
    You first need to install the VDS Powercli Fling:
    http://labs.vmware.com/flings/vdspowercli
    Seems like Alan actually forgot to link it with his “can be found here”.
    – Install the fling
    – you must also use the 32bit Powercli, since the VDS fling doesn’t work with the 64bit environment
    – Connect to your vCenter via “connect-viserver”
    – To get the function Alan wrote into your powershell session, execute “Import-Module D:\querysvmotionissue.ps1” or you might as well paste the whole code just into the shell, quick and dirty
    – Execute the Test-VDSVMIssue function against your VMs like “Get-VM | Test-VDSVMIssue”
    – Enjoy

  23. Jon Walmsley

    Hi Apologie for the basic question I am about to ask.

    How do i install/runt he script. I see you are calling a function from the screenshot, how am i able to add the script so I can call the function.

    Again sorry for the basic question, pretty new to PS and CLI

  24. Pingback: Scripts release for Storage vMotion / HA problem - Yellow Bricks

  25. Pingback: HA fails to initiate restart when a VM is SvMotioned and on a VDS! - Yellow Bricks

  26. Pingback: - Cliff Davies

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.