Category Archives: ESX/ESXi

Citrix on VMware, 1vCPU or 2vCPU ?

Following my previous Citrix testing on VMware I thought it best to try a dual vCPU VM to see if I can get double the users of a single vCPU, sounds logically sensible to me !

So just a reminder that a single vCPU with 25 users ran at around 73% average, a nice figure to run at giving plenty of CPU for peaks and troughs.

This test of a dual vCPU was undertaken following the sucesfull test of 25 users on one VM with 1 x vCPU this was run as a baseline to see how it relates to the single vCPU VM.

Virtual Machine Specification

2 x vCPU (Using VMware process affinity to tie the VM to a specific CPU core)
3072 GB RAM (reduced from 4GB as memory was not being used)

1 x 20GB Hard Disk Stored on SAN

1 x vNic (1GB)

Users Logged on: 50

  • Logged 10 users on and working over 5 minutes
  • Logged another 10 users on and working over 5 minutes
  • Logged another 5 users on and working over 3 minutes
  • Kept 25 users logged on and working for 15 minutes
  • Logged all 25 users off over a 5 minute period

clip_image002

As you can see from the above, a 3000Mhz server runs at around 60% which is not a good start bearing in mind the single vCPU VM.

Next Test 50 Users and Dual vCPU

This test was undertaken following the sucesfull test of 25 users on one server with 1xVCPU allowing for a reasonoble amount of capacity for application usage.

Virtual Machine Specification

2 x vCPU (Using VMware process affinity to tie the VM to a specific CPU core)
3072 GB RAM

1 x 20GB Hard Disk Stored on SAN

1 x vNic (1GB)

Users Logged on: 50

  • Logged 20 users on and working over 5 minutes
  • Logged another 20 users on and working over 5 minutes
  • Logged another 10 users on and working over 3 minutes
  • Kept 50 users logged on and working for 15 minutes
  • Logged all 50 users off over a 5 minute period

clip_image002[7]

The peak CPU usage of this graph was reached before 50 users were connected and therefore proves that the best method to get more users per physical server is to scale out rather than increase CPU resources.

This will obviously make sense in some cases where the cost of Operating System, Application software and licenses and management costs may make sense to build a 2 x vCPU and get slightly more users.

 

Unable to format a LUN

The SAN team issued a new LUN to my hosts and I was unable to format it as I recieved the following error: “Unable to read partition information” on LUN 180 on vmhba1.

Running fdisk from the service console cleared this up, heres my output :

[root@myhost]# fdisk /vmfs/devices/disks/vmhba1\:0\:180\:0

Command (m for help): p

Disk /vmfs/devices/disks/vmhba1:0:180:0 (Sun disk label): 15 heads, 64 sectors, 24686 cylinders
Units = cylinders of 960 * 512 bytes

Device Flag Start End Blocks Id System
/vmfs/devices/disks/vmhba1:0:180:0p3 u 0 24686 11849280 5 Whole disk
/vmfs/devices/disks/vmhba1:0:180:0p4 u 0 2 960 f Unknown
/vmfs/devices/disks/vmhba1:0:180:0p5 u 2 24686 11848320 e Unknown

Command (m for help): o
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won’t be recoverable.

The number of cylinders for this disk is set to 24686.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

After this I was able to use the LUN as normal, really put here for my reference in the future 🙂

Is MSCS Supported on a HA/DRS Cluster ?

(Straight answer in red at the bottom if you want to miss out the novel)

I needed to install Microsoft Cluster Service (MSCS) on a couple of virtual machines acting as an Active-Passive cluster, with a rule to separate these VM’s onto different hosts within VMware 3.5 U1.

Some may ask why (as I did) when you have the wonders that are HA and Vmotion why would you do this ? the short answer was that it took less time (slightly) for the MSCS cluster to fail over to another virtual machine on a different host in the event of hardware failure than it did for HA to realise there was a failure and move the machine and power it back on and the machine to start up.

And as this was a urgent application the MSCS cluster on top of VMware seemed to be the way to go.

So after reading the fantastic white paper by VMware:

Setup for Microsoft Cluster Service
Update 2 Release for ESX Server 3.5, ESX Server 3i version 3.5, VirtualCenter 2.5

Which can be downloaded here

Unfortunately I hit upon a problem in my planned design, on Page 16 of the above document which summarises the caveats for setting up a MSCS cluster on VMware, one of the bullet points was as follows:

  • Clustered virtual machines cannot be part of VMware clusters (DRS or HA)

I will not use the exact language here but lets just say I was not best pleased !

On thinking about this and the way that it worked and after adding a post on the VMware communities to confirm what I was thinking (great bunch of guys on there) I came up with the following…

As the MSCS Cluster has a private network which it uses for a heartbeat LAN there may be issues VMotioning the server over to another host, this as we know can cause a slight blip in the network communication and if this was to happen it may cause a problem with the MSCS cluster causing it to issue a failover and suddenly the service would be unavailable.

OK it makes sense, but that is quite a broad statement to make, “Clustered virtual machines cannot be part of VMware clusters (DRS or HA)”. Why couldn’t I just edit the DRS Cluster and set the Virtual Machines Automation Level to Disabled ? Surely this would stop the machines from being VMotioned and everyone would be happy ?

Whilst in theory this would work forever and a day and I would not get any issues I decided to confirm why the document did not reflect this setup as even if I did get it working it was point blankly “Not Supported” by VMware, and as we know working in a corporate environment this is a big NO NO.

Next Step: I raised a call with VMware to ask why this was the case, after a few games of Ping Pong I received my answer…..

Hi Alan,

You are correct in your assumption. It is not recommended to have MSCS VM’s within a Vmware Cluster. This is for the reasons you mentioned above regarding interrupted heartbeats. You could set individual HA/DRS settings on these individual MSCS VM’s in order to keep them running on the specific hosts. However, there would be issues in the future if these values were ever changed unknowingly by another administrator with access to the system.

Yes, I can confirm that this would be supported by VMware. However, if DRS ever tried to move these VM’s even with the DRS setting disabled then this would be a bug and would need to be investigated.

I would recommend that you test this setup using a network monitoring tool such as IO Meter to ensure that no heartbeats are dropped from the VM.

Correct HAL Installed ?

Following an inspiring blog post from Duncan Epping located here it made me think about how I could find the CPU vs HAL information, a 5 minute mess around in the VI Toolkit produced the following script:

Connect-viserver MYVISERVER

$myCol = @()

ForEach ($VM in (Get-VM)){
$MyDetails = “” | select-Object Name, HAL, NumvCPU
$MYDetails.Name = $VM.Name
$Hal = Get-WmiObject -ComputerName
$VM.Name -Query “SELECT * FROM Win32_PnPEntity where ClassGuid = ‘{4D36E966-E325-11CE-BFC1-08002BE10318}'” | Select Name
$MYDetails.HAL = $Hal.Name
$MYDetails.NumvCPU = $VM.NumCPU
$myCol += $MYDetails}

$myCol

This will give you a list of your VM’s the HAL installed and the number of vCPU’s

VMWorld 2009 VCP Special Prices

Just a reminder that if you are looking to attend VMWorld 2009 and you are a VCP you are eligible for a discounted price of 650 Euros….

http://www.vmworld.com/community/conferences/europe2009/registration/cost

VMworld VCP

VMworld VCP members are eligible for a discounted Full Conference rate of € 650. This discount will automatically be applied when you provide your VCP membership number, along with a valid email address, during the registration process. If the appropriate discount is not applied please contact VMworld Europe 2009 Registration Team. VCP pricing is not subject to enrolment date and will be valid throughout the enrolment period.

VMware Communities Roundtable

To listen to the Podcast being recorded live and experience all the fun of the chat room attend this weeks VMware Communities Roundtable;

Date: 12th November 2008

Time: 08:00 GMT – 15:00 EST

VMware experts talk about VMware news, interesting virtualization posts, and what’s happening in the VMware Communities.

Check here for a countdown to the podcast and to join closer to the time.

Virtualising Citrix/XenApp on VMware

As part of some investigation into virtualising Citrix servers I decided to evaluate the use of Windows 2003 32 bit Standard servers running as virtual machines on VMware ESX. The reasoning behind the use of virtual machines is to try and issue more Citrix sessions to a single server therefore reducing the amount of hardware, data center space, power consumption and heat generated by the multiple physical servers.

Moving to VMware should allow for multiple machines to run on a physical server, allowing for a higher user load by adding more smaller loaded servers to the physical server.

This obviously depends on how Citrix is used in your environment, as Citrix applications that are published to users who open the application and minimise it until they need the app, using there desktop applications most of the time will have much a lower utalisation impact than users who use full Citrix desktops.

It will also depend on the kind of applications that you are using and how CPU/Memory intensive these applications are.

Test Platform

In order to test this, and other environments, I have been using Citrix Edgesight for Load Testing. Edgesight for Load Testing allows me to develop “scripts” that simulate a user load. These scripts are made up of various user interactions, such as use of the Office products, and can then be replayed with various time parameters to simulate a real world load. Each script can be run in the context of a user session (separate users for each session) and so allowing us to get somewhere near generating a “real world” load.

The generated script is run against a specified server and the times taken to perform each action can be measured along with the built in Virtual Infrastructure performance statistics and graphs.

Physical Server (ESX Host) Specification

  • DELL PowerEdge 2950 III
  • 2 x Quad Core Intel Xeon X5450, 2X6MB Cache, 3.0GHz, 1333MHz FSB
  • 32GB FB 667MHz Memory (8x4GB dual rank DIMMs)
  • 2 x Intel PRO 1000VT Quad Port Gigabit Network Card, PCI-E
  • Emulex LPe-12002-E, Dual-port 8Gbps Fibre Channel PCI-Express HBA card
  • 2 x 73GB 15.000 rpm 3.5-inch SAS Hard Drives
  • ESX 3.5 Update 1

Virtual Machine Specification

  • 1 x vCPU (Using VMware process affinity to tie the VM to a specific CPU core)
    4GB RAM
  • 1 x 20GB Hard Disk Stored on SAN
  • 1 x vNic (1GB)

Virtual Machine Software Specification

Windows 2003 R2 32 Bit Standard Edition SP1
Citrix Presentation Enterprise 4.0 Rollup pack PSE400W2K3R03 plus additional hotfixes
Office 2003 with latest Service Pack

Other Configuration

The Virtual machines, ESX Hosts and VMTools were configured specifically to adjust the virtual machines for best performance of Citrix using multiple best practice and configuration documents.

Test Script

The test script was made up of 3 core sections that invoked the use of the following software:

  • Outlook 2003
  • Word 2003
  • Excel 2003

For each of the scripts the programs would be opened, used (either opening email or opening documents), and then left open and the next program opened. Once the Excel test had finished the test closed all of the Office applications and started again with Outlook. This was repeated until the allotted test time was completed and the user was then logged off.

Test Load

These tests were carried out with a number of users per virtual machine. The virtual machine was configured with 1 x vCPU to maximise the number of virtual machines that could be hosted on each host, as the host has 8 cores each core could host a virtual machine.


Test 1 – 10 Users

Test

Time Period

Users Logged On

Details

1

20 minutes

10

· Logged 10 users on and working over 5 minutes

· Kept 10 users logged on and working for 15 minutes

· Logged 5 users off over a 5 minute period

clip_image002


Test 2 – 20 Users

Test

Time Period

Users Logged On

Details

2

25 minutes

20

· Logged 10 users on and working over 5 minutes

· Logged another 10 users on and working over 5 minutes

· Kept 20 users logged on and working for 15 minutes

· Logged all 20 users off over a 5 minute period

clip_image004


Test 3 – 25 Users

Test

Time Period

Users Logged On

Details

3

33 minutes

25

· Logged 10 users on and working over 5 minutes

· Logged another 10 users on and working over 5 minutes

· Logged another 5 users on and working over 3 minutes

· Kept 25 users logged on and working for 15 minutes

· Logged all 25 users off over a 5 minute period

clip_image006


Test 4 – 30 Users

Test

Time Period

Users Logged On

Details

4

35 minutes

30

· Logged 10 users on and working over 5 minutes

· Logged another 10 users on and working over 5 minutes

· Logged another 10 users on and working over 5 minutes

· Kept 30 users logged on and working for 15 minutes

· Logged all 30 users off over a 5 minute period

clip_image008


Test 4 – 35 Users

Test

Time Period (Minutes)

Users Logged On

Details

5

38 minutes

35

· Logged 10 users on and working over 5 minutes

· Logged another 10 users on and working over 5 minutes

· Logged another 10 users on and working over 5 minutes

· Logged another 5 users on and working over 3 minutes

· Kept 35 users logged on and working for 15 minutes

· Logged all 35 users off over a 5 minute period

clip_image010


Results

During the tests CPU/Memory/Disk and Network figures were monitored, as the users were added to the virtual machines it was clear the application set used for testing were CPU bound and although all other figures were consistently monitored these would not take as heavy an impact as CPU.

From the above graphs we can see where the users were logging into the virtual machines and where the sessions conducted there test patterns, after an initial spike in the login period the figures averaged out at the following:

  • 10 Users – Averaged at 40% CPU Usage
  • 20 Users – Averaged at 65% CPU Usage
  • 25 Users – Averaged at 73% CPU Usage
  • 30 Users – Averaged at 81% CPU Usage
  • 35 Users – Averaged at 87% CPU Usage

Conclusion

Based on this testing and the environment we used, 25 users per single vCPU guest seems feasible (possibly a few more) but my opinion 30+ users is pushing sensible extended period CPU load due to resources needing to be made available for the tasks which will use the CPU for larger and longer periods of time which may impact the other running users of that particular machine.

If we extrapolate this out, it could mean 200+ (25 * 8) sessions on 8 guest machines across the 8 processor cores of the Dell 2950.

Please note that the current version of ESX at the time of writing this document is ESX 3.5 Update 3. As VMware continue to improve memory and CPU handling techniques it is feasible that further tests on this version may provide more sessions per virtual machine.

Unfortunately I was unable to get anywhere near the numbers that VMware achieved in there scalability document.

If you have any experience in virtualising Citrix or if you have any further tweaking or tuning tips please leave a comment below.

Part 2: Citrix on VMware, 1vCPU or 2vCPU ?

VMware ESX Server 3.5 Update 3 Released

Download it here

What’s New

The following information provides highlights of some of the enhancements available in this release of VMware Infrastructure 3:

Note: Not all combinations of VirtualCenter and ESX Server versions are supported and not all of these highlighted features are available unless you are using VirtualCenter 2.5 Update 3 with ESX Server 3.5 Update 3. See the ESX Server, VirtualCenter, and Virtual Infrastructure Client Compatibility Matrixes for more information on compatibility.

New features and supported IO devices:

  • Increase in vCPU per Core Limit — The limit on vCPUs per core has been raised from 8 (or 11 for VDI workloads) to 20. This change only raises the supported limit but does not include any additional performance optimizations. Raising the limit allows users more flexibility to configure systems based on specific workloads and to get the most advantage from increasingly faster processors. The achievable number of vCPUs per core will depend on the workload and specifics of the hardware. It is expected that most deployments will remain within the previous range of 8-11 vCPUs per core. For more information, see VI3 Performance Best Practices and Benchmarking Guidelines.
  • HP BL495c support — This release adds support for the HP Blade Server BL495c with all Virtual Connect and IO Options allowing 1 or 10Gb connection to the network (upstream) and 1Gb connections only to the servers (downstream).
  • Newly Supported NICs — This release adds support for the following NICs:
    • Broadcom 5716 1Gb
    • Broadcom 57710 10Gb Adapters
    • Broadcom 57711 10Gb Adapters at 1Gb speed only

    Note: iSCSI/TOE hardware offloads available with these adapters are not supported by VMware with ESX 3.5.

  • Newly Supported SATA Controllers— This release adds support for the following SATA controllers:
    • Broadcom HT1000 (supported in native SATA mode only with SATA hard drives and Solid State Disk devices)
    • Intel ICH-7 (supported in IDE/ATA mode only with SATA CD/DVD drives)

    Note: Storing VMFS data stores on drives connected to these controllers is not supported

  • Newly Supported Guest Operating Systems — Support for the following Guest Operating Systems have been added by VMware during the ESX 3.5 Update 3 release cycle:
    • Solaris 10 U5
    • Ubuntu 8.04.1
    • RHEL 4.7
  • Internal SAS networked storage controllers — This release adds experimental support for Intel Modular Server MFSYS25 SAS Storage Control Modules (SCMs). For known issues with this platforms and workaround see SAS Link and Port Failovers with the Intel Modular Server Running Update 3 and Later Versions of ESX 3.5 and ESXi 3.5
    (KB 1007394).
  • Interrupt Coalescing (IC) for Qlogic 4Gb FC HBAs — Introduced in this release, the feature reduces CPU utilization (and CPU cost per IO) and improves throughput of IO intensive workloads by generating a single interrupt for a burst of Fibre Channel frames, when received in a short period of time, rather than interrupting the CPU each time a frame is received. The feature is enabled by default.
  • Experimental Support for the VMDK Recovery Tool — This release adds support for the VMDK Recovery tool, a script intended to help customers to recover VMFS/vmdk data stores from accidental deletion of VMFS/vmdk data store or physical disk corruption. For more information, see VMDK Recovery Tool (ESX 3.5 Update 3) ( KB 1007243).
  • Small Footprint CIM Broker — Updated SFCB to version 1.3.0
  • IBM SAN Volume Controller — SVC is now supported with Fixed Multipathing Policy as well as MRU Multipathing Policy.

ESX4 in Workstation 6.5.1

I had tried this already as I don’t have any spare hardware at the moment but I just wanted to say a massive thanks to Eric, our test systems will live on !

A few weeks ago I filled a support request, because I couldn’t start a virtual machine on ESX4 while ESX4 was hosted on Workstation 6.5.0. ESX4 kept crashing at 95% during the VM Power On. An anonymous source just brought me up-to-date and whispered the following:

The WS 6.5 bug you noted earlier has been fixed. You will be able to run nested VMs under ESX 4.0 under WS 6.5.1, subject to the usual caveats (hardware virtualization has to be supported on your host for running the outer guest, and the inner guest is restricted to 32-bits). I can’t comment on release schedules, but it is only a dot-dot release, so it can’t be too far off.

Original post: http://www.ntpro.nl/blog/archives/737-Running-ESX4-under-Workstation-6.5.1-will-be-supported.html