PowerCLI: Shutdown your Virtual Infrastructure

Imagine your Power intake to your rack has failed, imagine your UPS has kicked in but is about to run out of power, you need to quickly shut down all of your virtual infrastructure…. quick run !

Or, you could let PowerCLI do the work for you and help you safely shutdown your entire virtual infrastructure, you could even tell your UPS software that when it gets to a certain amount of battery life left that it needs to run this script to safely shut things down.

When I first wrote this script it didn’t work quite as expected, we can easily tell all our guests to shut down nicely through the guest shutdown feature but as I found, this doesn’t always work, for example, what happens if one of your machines is sat there on the following screen:

image

If that happens then the guests never actually shut down and we are left in limbo waiting for the guests to “safely” be shut down, As you will see from the below script, I have added not only a check to see if the VMs are powered off but also a fail safe time where it just goes for it and shuts down the hosts anyway.

Connect-VIServer MyVIServer

# Get All the ESX Hosts
$ESXSRV = Get-VMHost

# For each of the VMs on the ESX hosts
Foreach ($VM in ($ESXSRV | Get-VM)){
    # Shutdown the guest cleanly
    $VM | Shutdown-VMGuest -Confirm:$false
}

# Set the amount of time to wait before assuming the remaining powered on guests are stuck
$waittime = 200 #Seconds

$Time = (Get-Date).TimeofDay
do {
    # Wait for the VMs to be Shutdown cleanly
    sleep 1.0
    $timeleft = $waittime - ($Newtime.seconds)
    $numvms = ($ESXSRV | Get-VM | Where { $_.PowerState -eq "poweredOn" }).Count
    Write "Waiting for shutdown of $numvms VMs or until $timeleft seconds"
    $Newtime = (Get-Date).TimeofDay - $Time
    } until ((@($ESXSRV | Get-VM | Where { $_.PowerState -eq "poweredOn" }).Count) -eq 0 -or ($Newtime).Seconds -ge $waittime)

# Shutdown the ESX Hosts
$ESXSRV | Foreach {Get-View $_.ID} | Foreach {$_.ShutdownHost_Task($TRUE)}

Write-Host "Shutdown Complete"

This can be changed to only shutdown all vms and hosts in a certain datacenter of cluster by amending line 04 to the following:

For a specific datacenter, mine is called DC1…

$ESXSRV = Get-DataCenter “DC1” | Get-VMHost

For a specific cluster, mine is called Production…

$ESXSRV = Get-Cluster “Production” | Get-VMHost

I’m sure I don’t need to tell you that you need to be extremely careful when testing this script as one false move could shut down everything ! – Please test this to make sure it works first !

* Just to mention, this will obviously not work if your virtual center is a VM, for that you will need to do some funky connecting to each host etc, let me know if you desperately need that.

54 thoughts on “PowerCLI: Shutdown your Virtual Infrastructure

  1. Pingback: ESXi host shutdown with APC UPS connected via USB | Phy2Vir

  2. Pingback: How to shutdown all VMs in an ESXi host using PowerCLI | rtfmp

  3. Chakradhar Mishra

    I need you help to exclude vcenter and database server’, later connect to particular esx server and shutdown the vcenter and database server and particular esx server. Please help to tweak this scrip

  4. Chris

    What would you have to do in the case that your virtual center is an actual VM? I’m a network engineer by profession so I’m trying to teach myself how to use Powershell and PowerCLI on my own. Any help is greatly appreciated.

  5. pedro

    and if i wanted to set the ESX in maintenance mode first? would it be better, and how to do it??

    worked for me like a charm!

    Regards

  6. Pingback: Graceful shutdown of an ESXi 5.1 host and guest VMs (free edition) using the shell/command line/scripting (UPS friendly) | Open Enterprises

  7. Eduard

    hello,

    your sctipt is very interesting. I also have a similar solution. My vCenter enables a VM and I do not know how to turn off the infrastructure. Maybe you have an idea?

  8. Lorenc

    I had the same problem whit the loop so I changed all .seconds to .TotalSeconds
    and it works.

    Lorenc

  9. Pingback: How do I shutdown the Host over ssh on ESXi 5 so it shuts down the guests properly? - Admins Goodies

  10. Jim Nickel

    I had a similar problem. I changed my loop to the following which works fine:


    $waittime = 180 #Seconds
    do {
    # Wait for the VMs to be Shutdown cleanly
    sleep 10.0
    $waittime = $waittime - 10
    $numvms = @(Get-VM | Where { $_.PowerState -eq "PoweredOn" }).Count
    Write "Waiting for shutdown of $numvms VMs or until $waittime seconds"
    } until ((@(Get-VM | Where { $_.PowerState -eq "PoweredOn" }).Count) -eq 0 -or $waittime -le 0)

    Jim

  11. arran

    Thanks for this script, it is alll working apart from the countdown.. It starts at 200 then once it gets to 140 (ie 60 seconds) it loops back to 200. The only thing i changed was the $ESXSRV lines from

    $numvms = ($ESXSRV | Get-VM | Where { $_.PowerState -eq “poweredOn” }).Count
    Write “Waiting for shutdown of $numvms VMs or until $timeleft seconds”
    $Newtime = (Get-Date).TimeofDay – $Time
    } until ((@($ESXSRV | Get-VM | Where { $_.PowerState -eq “poweredOn” }).Count) -eq 0 -or ($Newtime).Seconds -ge $waittime)

    to $numvms = ($VMwareFolders | Get-VM | Where { $_.PowerState -eq “poweredOn” }).Count
    Write “Waiting for shutdown of $numvms VMs or until $timeleft seconds”
    $Newtime = (Get-Date).TimeofDay – $Time
    } until ((@($VMwareFolders | Get-VM | Where { $_.PowerState -eq “poweredOn” }).Count) -eq 0 -or ($Newtime).Seconds -ge $waittime)

    Where $VMwareFolders is a variable to for the folder containers of the VMs i want to shutdown
    foreach ($getVM in (Get-Folder -Name $VMwareFolders | Get-VM | Where {$_.PowerState -eq “poweredOn”}))

    The countdown results in this:

    Waiting for shutdown of 35 VMs or until 185 seconds
    Waiting for shutdown of 35 VMs or until 199 seconds
    Waiting for shutdown of 35 VMs or until 196 seconds
    Waiting for shutdown of 35 VMs or until 194 seconds
    Waiting for shutdown of 35 VMs or until 191 seconds
    Waiting for shutdown of 35 VMs or until 189 seconds
    Waiting for shutdown of 35 VMs or until 186 seconds
    Waiting for shutdown of 35 VMs or until 183 seconds
    Waiting for shutdown of 35 VMs or until 181 seconds
    Waiting for shutdown of 35 VMs or until 178 seconds
    Waiting for shutdown of 35 VMs or until 176 seconds
    Waiting for shutdown of 35 VMs or until 173 seconds
    Waiting for shutdown of 35 VMs or until 171 seconds
    Waiting for shutdown of 35 VMs or until 168 seconds
    Waiting for shutdown of 35 VMs or until 166 seconds
    Waiting for shutdown of 35 VMs or until 163 seconds
    Waiting for shutdown of 35 VMs or until 161 seconds
    Waiting for shutdown of 35 VMs or until 158 seconds
    Waiting for shutdown of 35 VMs or until 155 seconds
    Waiting for shutdown of 35 VMs or until 153 seconds
    Waiting for shutdown of 35 VMs or until 150 seconds
    Waiting for shutdown of 35 VMs or until 148 seconds
    Waiting for shutdown of 35 VMs or until 145 seconds
    Waiting for shutdown of 35 VMs or until 143 seconds
    Waiting for shutdown of 35 VMs or until 200 seconds
    Waiting for shutdown of 35 VMs or until 198 seconds
    Waiting for shutdown of 35 VMs or until 195 seconds

  12. Pingback: Shutdown von VMware ESXi-Hosts und VMs mit PowerCLI-Skript | blog.bistron.eu

  13. Jim Nickel

    Actually with a little help from Mike Foley, I have it figured out.

    Since APCUPSD is running as a service, the ExecutionPolicy doesn’t affect it, so you have to include it on the command line. Here is what worked for me:

    %SystemRoot%\system32\windowspowershell\v1.0\powershell.exe -psc “C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\vim.psc1” -NoLogo -NonInteractive -ExecutionPolicy RemoteSigned -Command “C:\UPS\PowerOffAll.ps1”

    The important piece is the -ExecutionPolicy RemoteSigned.

    Thanks,

    Jim

  14. XXXXX

    I am not at work until tomorrow, but from memory:

    I think I had to set the APC service with elevated permissions.

    Then I had to do a “double bat”. APC event “command” that calls a batch file to run a second batch file to start the powershell script.

  15. kim

    For starters, I think PowerCLI only has write access to ESXi.
    So not sure how you got it to work at all?

    Are you logging the output of the powershell script execution?
    If so, what does it say?

    I’m assuming the APCUPSD will be running under a different account than when you run the bat file manually?
    So permissions would be a place to start?

  16. Jim Nickel

    Hiya,

    I am using a modified version of your script to gracefully shutdown my vSphere 5 ESXi environment.

    I am using APCUPSD to monitor my UPS from a Windows 2008 R2 server.

    My problem is this – if I double click the .BAT file it works just fine (the Powershell runs and the ESXi VMs and hosts shutdown nicely), but the Powershell never gets run during an actual event, although the .BAT file IS run. The before.txt and the after.txt DO get created during an actual event, but the PowerShell does NOT happen.

    The Batch file looks like this:

    echo GOT HERE > c:\ups\before.txt

    %SystemRoot%\system32\windowspowershell\v1.0\powershell.exe -psc “C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\vim.psc1” “& C:\UPS\PowerOffAll.ps1”

    echo Got Here > c:\ups\after.txt

    Any ideas?

    Thanks,

    Jim

  17. Chad

    Nevermind — I had changed the wait time, and it was looping at 65 secs, I changed it again and all is well. Thank you for a great script!

    BTW, some may find it useful to use the VMWare credential store, as opposed to storing their credentials in a script, here’s the connection string I have based on this:

    http://communities.vmware.com/thread/199583

    C:\WINDOWS\system32\windowspowershell\v1.0\powershell.exe -PSConsoleFile “C:\Program Files\VMware\Infrastructure\VSphere PowerCLI\vim.psc1” -NoExit –Command
    $creds = Get-VICredentialStoreItem -Host ‘yourhost’
    Connect-VIServer -Server yourserver -User $creds.User -Password $creds.Password
    # Connect to the VC server credential store

  18. Pingback: Vmware ESXi shutdown « DefaultGateway

  19. Chad

    Hi, for those VMs that do not have MVtools, they will cause the script to wait, and not shut down the host. I’m setting up this script to trigger on heat or water in server room, so want things off, how would I set it so that it does a hard shut down of any VMs that have hung?

  20. Peter

    Thanks for very useful script. The only problem i had is VCENTER is VM. I did see post above about creating folders but this is what i did to exclude VCENTER until all VM’s down

    # For each of the VMs on the ESX hosts
    Foreach ($VM in ($ESXSRV | Get-VM | Where { $_.PowerState -eq “poweredOn” } | Where { $_.Name -ne “VCENTER” })){

  21. Anon

    I’m going round in circles with this script. I just want to stop all VM’s in a RP and not get a warning if they are already powered off. Any help much appreciated.

    # Stop all VM’s in a RP

    do

    {
    # Get the name of the Resource Pool
    $ResourcePool = read-host ” Insert the Name of the Resource Pool to Stop”
    # Get the members of that RP
    $VMName = (get-resourcepool $ResourcePool | get-vm)
    # For each of the VMs in the RP
    {
    Foreach ($VMName in ($ResourcePool | get-vm | Where-object {$_.PowerState -eq “poweredOn” }))
    }
    # Shutdown the guest cleanly
    {$VMName | Stop-VM -Confirm:$false}
    }
    until ($script:response -eq “quit”);

  22. XXXXX

    This is how I got around the VCenter as a VM problem:

    Start-Transcript -Path C:\CSS_Log.txt -Append

    Add-PSSnapin VMware.VimAutomation.Core
    Set-PowerCLIConfiguration -DefaultVIServerMode multiple -Confirm:$False

    Connect-VIServer HOSTA,HOSTB,HOSTC -User ‘USER’ -Password ‘PASSWORD’
    Get-VMHost | Get-VMHostStartPolicy | Set-VMHostStartPolicy -Enabled:$True -WaitForHeartBeat:$True
    Get-VM | Get-VMStartPolicy | Set-VMStartPolicy -StartAction PowerOn -UnspecifiedStartOrder
    Get-VM -Name SVR-SQL01 | Get-VMStartPolicy | Set-VMStartPolicy -StartAction PowerOn -StartOrder 3
    Get-VM -Name SVR-VC01 | Get-VMStartPolicy | Set-VMStartPolicy -StartAction PowerOn -StartOrder 2
    Get-VM -Name SVR-DC01 | Get-VMStartPolicy | Set-VMStartPolicy -StartAction PowerOn -StartOrder 1
    Get-VM | Shutdown-VMGuest -Confirm:$False
    DO {Get-VM | Get-VMGuest} WHILE (Get-VM | Get-VMGuest | where {$_.State -eq “Running”})
    Get-VM | Stop-VM -Confirm:$False
    DO {Get-VM} WHILE (Get-VM | where {$_.PowerState -eq “PoweredOn”})
    Get-VMHost | Stop-VMHost -Force -Confirm:$False

    Stop-Transcript

  23. lethalduck

    I’m not using $host as a variable.
    The PowerCLI admin manual on pg 14 is.
    As shown above.
    So obviously it’s another VMware error?

    Also be aware that any shutdown will not work on ESXi.
    As They have disabled write access to the host.

    So I’m in the process of writing a shutdown application to get around this new limitation and will be posting on my blog when done if anyone interested.

  24. Virtu-Al

    Yeah, you shouldnt use $host as a variable as this is a reserved variable, type $host in PowerShell and you will get information about the current powershell session, I would change the $host in your script to $h as you are then calling $h on the next line.

  25. cowboy

    @Virtu-Al

    thank you for your advice.i added the flowing code to my script
    if (!(get-pssnapin -name VMware.VimAutomation.Core -erroraction silentlycontinue)) {
    Write-Host “add-pssnapin …”
    add-pssnapin VMware.VimAutomation.Core
    }else{
    Write-Host “already exist.no need to add”
    }
    it shows “already exist.no need to add”.and no output from get-vm cmdlet.

  26. lethalduck

    Hi.

    I’m looking at shutting down a ESXi host with running guests.
    The guests should get automaticaly shutdown by the host if configured correctly.
    So just need to shutdown the host cleanly.
    Also need to shut down another FreeNAS box, but that’s another story.
    Been playing with a few scripts, then saw one in the vSphere PowerCLI Administration Guide on pg 14:
    Downloadable from here http://www.vmware.com/support/developer/PowerCLI/PowerCLI41U1/html/index.html
    5 Shut down the myHost virtual machine host
    $host = Get-VMHost myHost
    $hv = Get-View $h.ID
    $hv.ShutdownHost_Task($true)
    Any one got any idea what the $h is?
    Also when I run the first line in PowerGUI Script Editor I get:
    Cannot overwrite variable Host because it is read-only or constant.
    At C:\Users\user\AppData\Local\Temp\7aad3dc5-990e-4568-a06b-987092d00575.ps1:5 char:6
    + $host <<<< = Get-VMHost $vSphereServername
    + CategoryInfo : WriteError: (Host:String) [], SessionStateUnauthorizedAccessException
    + FullyQualifiedErrorId : VariableNotWritable

    Then for the second line I get:
    Get-View : Cannot validate argument on parameter 'VIObject'. The argument is null or empty. Supply an argument that is not null or empty and then try the
    command again.
    At C:\Users\user\AppData\Local\Temp\7aad3dc5-990e-4568-a06b-987092d00575.ps1:6 char:15
    + $hv = Get-View <<<< $h.ID

    I've already established connection to the host by way of Connect-VIServer.
    Anyone got any ideas?

  27. Virtu-Al

    try adding the following code to the top – it may be that you do not have the cmdlets loaded as part of the session:

    if (!(get-pssnapin -name VMware.VimAutomation.Core -erroraction silentlycontinue)) {
         add-pssnapin VMware.VimAutomation.Core
     } 
  28. cowboy

    i can run this script with double click the script file.but when i register it to taskschd.msc,it failed to get-vm:
    the return value of get-vm is empty !although the get-vmhost work perfectly.

    i register the script to taskschd.msc with checking “run whether user is logged on or not “.

    could tell why the return value of get-vm is empty ?thank you.

  29. Cassandra

    Looking for help on a script that will check all vms on a host and if only one VM is remaining running and that VM has a specific name, then that VM is shut down so that the DPM will suspend the host. Any ideas?

    Thanks!

  30. Pingback: Script of the Day – shutdown your VMware ESX estate with PowerCLI « A geek in the seat

  31. Pingback: Micronauts » Blog Archive » Script: Emergency – Shutdown

  32. Pingback: Micronauts » Blog Archive » Script: Emergency II – Shutdown

  33. Pingback: | Sys Admin Extraordinaire | » Blog Archive » Blogging Tips and Things That I Learned While Setting Up This Blog

  34. Pingback: script to shutdown guest & host ESXi

  35. shartma

    I am looking for a way to implement shutting down ESXi hosts using a VMA. I have tried ghetto shutdown script but it waits for each guest to shutdown before continuing to the next. Has anyone found a better solution for VMA and ESXi (paid)?

    thanks

  36. tomkus

    Hello,
    my answer to how to shutdown all VM’s expect vCenter or DC’s – create folder for all DC’s Machines, vCenter and rest
    and then first shutdown all machines “rest”

    first:
    Get-Folder -Name “rest” | Get-VM | Shutdown-VMGuest –Confirm:$false

    when script confirms all i shutdown then

    second:
    Get-Folder -Name “DC’s” | Get-VM | Shutdown-VMGuest –Confirm:$false

    and last

    Get-Folder -Name “vcenter” | Get-VM | Shutdown-VMGuest –Confirm:$false

    or if you want to shutdown all esx as well do not shutdown vcenter, – use powercli to shutdown all esx’s and set in esx’s poweroff option to shutdown all machines….

  37. Tom

    Have you had time/opportunity to figure out how this script could be adjusted to even partially work when the vCenter server is a VM??

    That is, perhaps one can/should set this up to work only with the hosts that do not contain the vCenter Server, and let the UPS etc. manage the host containing vCenter Server?? It could be assumed the SAN will simply power on…

    Or set things to shut down all VMs but vCenter and then go from there etc.??

    I am new to PowerShell and HA so I cannot yet offer much in this regard…I know most of us VMware admins stand on the shoulders of giants like you. 🙂 🙂

    Thank you, Tom

  38. Sebastian Kayser

    Great script! There is indeed not that much information on automated full environment shutdowns out there. Has one of the priority-based scripts already found its way to the public somewhere?

    Before using this script one might want to double-check for VMs with a missing VMware Tools installation. Just had a customer with mostly Linux VMs where many VMs simply didn’t have the Tools installed and thus the graceful guest shutdown wouldn’t work. Either use vCheck or something like following snippet:

    $VMS_WITHOUT_TOOLS = ( get-view -viewtype virtualmachine `
    -filter @{ “Guest.ToolsStatus” = “toolsNotInstalled”; “Runtime.PowerState” = “poweredOn” } `
    -property Name )

    if ($VMS_WITHOUT_TOOLS) {
    write-host “These VMs are missing VMware Tools, Shutdown-VMGuest won’t work for them”
    $TOOLS_MISSING_ON | %{ write-host “-” $_.Name }
    }

    Sebastian

    P.S.: Perl SDK users might want to check out William’s ghettoUPSHostShutdown.pl too: http://communities.vmware.com/docs/DOC-11902

  39. Jan

    Hello.
    I am new to the vmware and powershell. I have just a small family server which is still very important for me. I am using Vmware esxi. It is now set so that when I manually shutdown the host, it waits till guests get shutdown. The order which system is starting first and shutting last is set in configuratin. As I understand the script, it shutdowns the guests first. Is there any possibility just to tell esxi to shutdown so that it uses it’s own configuration? It must be set from Windows machine and it needs to be set a reaction on some event. Scheduled task is running and when some conditions will occure I need the task to send shutdown. Is it even possible?
    Thank you
    Jan

  40. Jesse

    In response #2 it was discused to have a priority setting of some kind to have some VM’s shutdown before others. How would one do this, we want to make sure out DC’s go down last.

    I have tested your script and it works great other then this one item, thanks.

    Jesse

  41. Ben

    Very cool! Only problem I came across is that we have lots of VM’s that are already shutdown or suspended. We also wanted to make the script suspend the VM’s instead of shutting them off. Pretty easy to implement both those changes:

    # For each of the VMs on the ESX hosts
    Foreach ($VM in ($ESXSRV | Get-VM | Where { $_.PowerState -eq “poweredOn” } )){
    # Suspend the guests cleanly
    $VM | Suspend-VM -Confirm:$false

  42. Pingback: vMA – Using HP Power Protector agent to shutdown virtual hosts | Daz's bits and bobs

  43. Chris Kyte

    Hi,

    I have been looking for something like this for a while, However I do not have vcentre and connect to my ESXi hosts directly. Unfortunately I have no PowerCLI experience. Can anyone offer any help as to how I would change this to talk to the ESXi hosts directly and not to Vcentre.

    Many Thanks

    Chris

  44. vishy

    Hi Alan,

    That will be great, I’ll look forward for an update script.

    Thanks and Regards,
    Vishy

  45. Virtu-Al

    @vishy

    Good Idea, I didnt think of using custom fields like that, mind if I write the script as a follow up post ?

    Alan

  46. vishy

    Nice script….I’m looking for a similar script to shutdown guest based on priority. I have a custom field called Priority attached to each guest with value 1,2 or 3. I would like to shutdown guest with low value 3 first followed by value 2 and last value 1

    Cheers,
    vishy

  47. Pingback: PowerCLI: Shutdown your Virtual Infrastructure | Expert Data Labs Blog

  48. Pingback: uberVU - social comments

  49. sketchy

    Great post! I’m amazed at how little this topic is addressed in posts and articles.

    Improvements could come in a few forms.

    1. It would be nice if there were a way to dry-run test it, so that you could do some validating without it kicking in.

    2. In reality, there is a general order in which one would want vm’s to gracefully shut down. It’s no fun if your DC’s shut down first.

    As for your comment about vcenter not being a VM in order for this to work, I think there is another way. One would have their cluster of ESX hosts. Then, you have a cheap old PC with a few extra NIC’s, with ESXi built up, connected to the SAN as well, but only running one VM; the server running vcenter. That way you have the flexibility of it being a VM on the SAN, but living outside the cluster.

Leave a Reply to Ben

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.