Backing up the ESXi System Image

ESXi is based on a system image, VMware used to call this a firmware but some people found this misleading, most people still refer to it as firmware but in this post I will refer to it as a system image.

The system image is a unified image which is the same whether booting from USB, Hard Disk, PXE or any other media. The logic in the first boot will provide auto configuration based on the kind of installation you have.

One thing to remember about ESXi is that it is memory based, so once booted the system image is entirely loaded into memory, ESXi doesn’t care if the original media disappears after boot, there is no reliance on the boot device for running after booting.

ESXi is essentially made up from a number of tardisks (VM TAR files) or archives, these tardisks are mounted into the empty root filesystem in order of enumeration, one of these is called the state tardisk or state.tgz.

The state tardisk is made up of any file in /etc which is marked as sticky, VMware uses the sticky bit to flag the files which are to be included as part of this tardisk but only if they reside under /etc, these are files like esx.conf, inetd.con, passwd….  These are the files which will be persisted after boot.

As said earlier the entire system image is loaded into memory on boot so how is the state tardisk persisted ?

Periodically there is a deamon in the background which looks for changes in the config files, when these are found the state tardisk is rebuilt and saved back to the bootdisk, so why are these files not written directly to disk each time ?  The main reason is the USB disk, if these files were written constantly to the USB flash disk then the life expectancy of the disk would not be very long at all, VMware worked with a number of USB flash OEM’s to calculate the best frequency to write these files back to disk.  This is key to remember as the frequency is around once every 10 minutes, obviously if the system crashes and reboots you may loose any configuration which may have been made in the previous 10 minutes.

So knowing this we can effectively back up the configuration of our ESXi hosts (- 10 minute changes worst case) simply by backing up the state tardisk, then if the worst happens we can restore this and have our ESXi host with complete configuration restored in a matter of minute.  Luckily PowerCLI makes this very easy for us, the below script can be run as a scheduled task and will backup the state tardisk as often as the scheduled task is run.

$RootFolder = "C:\Support\"
Get-VMHost | Foreach {
	Write-Host "Backing up state for $($_.Name)"
	$Date = Get-Date -f yyyy-MM-dd
	$Folder = $RootFolder + $Date + "\$($_.Name)\"
	If (-not (Test-Path $Folder)) {
		MD $Folder | Out-Null
	}
	$_ | Get-VMHostFirmware -BackupConfiguration -DestinationPath $RootFolder
	# Next line is a workaround for -DestinationPath not working correctly
	# with folder names with a - in them.
	MV ($RootFolder + "*") $Folder -ErrorAction SilentlyContinue
}

image

Research has been gathered for this post from TA8245 – ESXi Internals by Oliver Cremel

28 thoughts on “Backing up the ESXi System Image”

  1. This script does everything that it is intended to do, thanks.. One question is there an easy way to restrict the number of previous backups to say 7. This would limit the manual intervention needed to purge old backups?

  2. Hello ,

    Thank for that script It helps me a lot.

    I modify it a little bit because I have more than 140 ESX in my VC and I want to backup some of them.

    Here is the script with “hostlist file”

    ##### Start

    $RootFolder = “C:\Users\vhristev\Desktop\ESX_backup_dmz\”
    Get-VMHost -Name (Get-Content C:\Users\vhristev\Desktop\hostlist.txt) | Foreach {
    Write-Host “Backing up state for $($_.Name)”
    $Date = Get-Date -f yyyy-MM-dd
    $Folder = $RootFolder + $Date + “\$($_.Name)\”
    If (-not (Test-Path $Folder)) {
    MD $Folder | Out-Null
    }
    $_ | Get-VMHostFirmware -BackupConfiguration -DestinationPath $RootFolder
    # Next line is a workaround for -DestinationPath not working correctly
    # with folder names with a – in them.
    MV ($RootFolder + “*”) $Folder -ErrorAction SilentlyContinue
    }

    ### END

  3. Virtu-Al

    William? I don’t know either… I guess too much multi-threading.. :S
    When I run Get-VMHost I only get the three servers, but in a formatted table with columns.

    1. The strange thing is William is my middle name !

      I dont understand why it is running more than he amount of hosts you have as all im doing is saying get a list of hosts and then for each one export the firmware, very strange !

  4. William,

    I’m wondering why do I get seven output lines when I only have three hosts.
    The ‘foreach’ statement seems to run seven times instead of three (for each host).
    The loop goes two times through each host and then one third time through the first before it ends.
    This means it’s writing twice the second and third host’s backup and three times the first.
    Any ideas? I’m using the $env:Temp path, no sub-folders and no dates. Also I’m not using the MV bug line.

  5. I never even thought of backing up the System Image on my ESXi servers. In case of a problem, I just do a fresh install and run my configuration PowerCLI script against the new/restored server, which resets all my parameters and configuration to a “known good” state.

  6. @Nigel,

    Both vicfg-cfgbackup & Get-VMHostFirmware stem from the same vSphere API .. so there is no advantage using one or the other. It just depends on whether you’re trying to do this in PowerCLI or vCLI or even another vSphere SDK

    Backup API Method – http://www.vmware.com/support/developer/vc-sdk/visdk41pubs/ApiReference/vim.host.FirmwareSystem.html#backupConfiguration

    Restore API Method – http://www.vmware.com/support/developer/vc-sdk/visdk41pubs/ApiReference/vim.host.FirmwareSystem.html#restoreConfiguration

    I actually wrote a vSphere SDK for Perl script awhile back that will keep multiple backups in which you can easily backup and restore from – http://communities.vmware.com/docs/DOC-11767

    I’m sure it’s trivial for Alan to provide something of that nature and restoring the ESXi config is also done using same vSphere APIs, I’ll defer #2 to Alan for a PowerCLI

    –William

  7. Tried your script and it bombed out as shown below:

    Backing up state for esx1.blahblahblah.local
    Get-VMHostFirmware : A parameter cannot be found that matches parameter name ‘B
    ackupConfiguration’.
    At C:\Build\BackupESXIImage.ps1:10 char:49
    + $_ | Get-VMHostFirmware -BackupConfiguration <<<< -DestinationPath $Root
    Folder

    I'm running vSphere 4.1 but have not yet applied Update 1. Am I missing something obvious?

  8. Hi Al,
    A couple of questions:
    1) is this any better than using vicfg-cfgbackup?
    2) How do you do a restore and will the ESXi host automatically pick up the config from the tarball rather than overwriting it with its blank running config?

    1. Both fantastic questions, if you dont mind I will answer them in a further blog post, too much to say to add it to a comment and have people miss it, thanks for the great questions.

Leave a Reply