Maximizing VMware Performance and CPU Utilization

February 22, 2017 | Heroix Staff

In a previous post we discussed overcommitting VMware host memory – the same can be done with host CPU. As per the Performance Best Practices for VMware vSphere 6.0:

In most environments ESXi allows significant levels of CPU overcommitment (that is, running more vCPUs on a host than the total number of physical processor cores in that host) without impacting virtual machine performance. (P. 20, ESXi CPU Considerations)

This post will discuss calculating CPU resources, considerations in assigning resources to virtual machines (VMs), and which metrics to monitor to ensure CPU overcommitment does not affect VM performance.

Calculating available Host CPU resources

The number of physical cores (pCPU) available on a host is calculated as:

(# Processor Sockets) X (# Cores/Processor)  = # Physical Processors (pCPU)

If the cores use hyperthreading, the number of logical cores is calculated as:

(# pCPU) X (2 threads/physical processor) = # Virtual Processors (vCPU)

For example:

4 sockets X 6 cores/processor = 24 pCPU
24 pCPU X 2 threads/core = 48 vCPU

Please note that hyperthreading does not actually double the available pCPU. Hyperthreading works by providing a second execution thread to a processor core. When one thread is idle or waiting, the other thread can execute instructions. This can increase efficiency if there is enough CPU Idle time to provide for scheduling two threads, but in practice performance increases are up to a maximum of 30% and are strongly application dependent.

Considerations in Assigning vCPUs to VMs

1. Best Practices recommendations

  • Start with one vCPU per VM and increase as needed.
  • Do not assign more vCPUs than needed to a VM as this can unnecessarily limit resource availability for other VMs and increase CPU Ready wait time.
  • The exact amount of CPU overcommitment a VMware host can accommodate will depend on the VMs and the applications they are running. A general guide for performance of {allocated vCPUs}:{total vCPU} from the Best Practices recommendations is:

    • 1:1 to 3:1 is no problem
    • 3:1 to 5:1 may begin to cause performance degradation
    • 6:1 or greater is often going to cause a problem

2. Non-Uniform Memory Architecture (NUMA)

In a previous post on minimizing CPU latency with NUMA, we discussed performance degradation on multiprocessor VMs in which the number of vCPUs was greater than the number of vCPUs in a NUMA node. Generally, try to keep multiprocessor VMs sized so that they fit within a NUMA node.

3. Co-stop

VMware schedules all the vCPUs in a VM at the same time. If all the allocated vCPUs are not available at the same time, then the VM will be in a state of "co-stop" until the host can co-schedule all vCPUs. In its simplest form co-stop indicates the amount time after the first vCPU is available until the remaining vCPUs are available for the VM to run. Sizing VMs to use the least number of vCPU’s possible minimizes the time needed for co-stop waits.


Monitoring VMware CPU metrics

Monitor the following metrics to fine tune the number of vCPUs allocated per VM and to ensure that CPU overcommitment does not degrade performance:

1. VM CPU Utilization

Monitor CPU Utilization by the VM to determine if additional vCPUs are required or if too many have been allocated. CPU use can be monitored through VMware or through the VM’s operating system. Utilization should generally be <= 80% on average, and > 90% should trigger an alert, but this will vary depending on the applications running in the VM.

2. VM CPU Ready

VM CPU Ready is a measure of the time a VM has to wait for CPU resources from the host. VMware recommends CPU ready should be less than 5%.


Longitude high CPU Ready VMware report
Figure 1: Longitude Report of VMware showing high CPU Ready values.


3. Co-Stop

Applicable to VMs with multiple vCPUs – is a measure of the amount time after the first vCPU is available until the remaining vCPUs are available for the VM to run. A co-stop percent that is persistently >3% is an indication that a right-sizing exercise may be in order.


esxtop showing high co-stop
Figure 2: esxtop showing a VM with a high co-stop value.


4. VMware host CPU Utilization

Monitor CPU Utilization on the VM host to determine if CPU use by the VMs is approaching the maximum CPU capacity. As with CPU usage on VMs, CPU utilization at 80% - 90% should be considered a warning level, and >= 90% indicates that the CPUs are approaching an overloaded condition.


Longitude high host CPU report
Figure 3: Longitude Capacity Planner showing host CPU approaching capacity.



Overcommitting CPU allows you to maximize use of host CPU resources, but make sure to monitor overcommitted hosts for CPU use, and CPU Ready and Co-stop percentages. Avoid oversizing VMs with more vCPU’s than needed. Consider NUMA architecture and the effect of co-stop waits when creating VMs with multiple vCPUs.

Want to learn more?

Download our Virtualization or Cloud IaaS Whitepaper - both technologies can provide redundancies that will maximize your uptime and that will allow you to squeeze out the most performance. Which is better and how do you decide?

Download the whitepaper:  Virtualization or Cloud IaaS?