Finding and correcting performance bottlenecks in a vSphere lab

A VMware lab is a great way to test upgrades and new products or just to learn about vSphere, but you’ll find little benefit to a lab hampered by performance bottlenecks.

Without even analyzing performance statistics, I can tell you that the most common issue in vSphere labs is slow VM response. The slow VM could be your vCenter virtual machine (VM) or other test VMs that you are running.  Depending on your needs, you may want to boost performance of one or the other of these VMs (or both).

Poor VMware lab performance usually boils down to one of two things:

• Too little vRAM: When a VM is running slowly, the first thing I check is the amount of vRAM assigned to it. Yes, Windows XP with no applications runs fine with 1 GB of vRAM, but a Windows 2008 R2 64-bit VM running vCenter and the local SQL Express database runs like a pig with just 1 GB of RAM. It's easy to forget the needs of the applications running in the VM when sizing. Remember that a generic install of Windows 2008 requires a lot less vRAM than that same VM with applications installed and running.
• Storage latency: High performance storage with lots of spindles and cache is expensive. Whether it's production or lab, we tend to undersize storage more than any other resource. VSphere has advanced features that it can use when memory is constrained, but when storage I/O is constrained, there isn't much that an ESXi host can do to help. The host needs the blocks of the Virtual Machine Disk file format from the shared storage and, if it can't get it, that VM will appear slow or frozen, even if you are just opening the calculator application.

Using tools to identify performance bottlenecks
In order to use statistics to identify the performance bottlenecks in a vSphere lab, you would go through the same process that you would in a production environment. For example, you could begin by looking for CPU read values (A value of more than 10% in esxtop indicates CPU contention), instances of memory swapping, memory being reclaimed by ballooning and disk device latency (anything more than 25 milliseconds is considered bad in production, but you may be more tolerant in a lab).

What tools can help you? Of course, the tried and tested vSphere command-line interface tool is esxtop, and runs on an ESXi server console. You can also use your vSphere client's performance graphing tools to analyze similar performance stats.

Call me lazy, but I enjoy using some of the newer graphical performance tools. There are free versions of most commercial performance tools and some vendors will even give you a full commercial license if you only use it in your lab environment. Here are three of my favorites:

VKernel Corp.’s vOPS Server Explorer is a free tool is deployed as a virtual appliance. It shows you a green, yellow or red indicator for each ESXi host and VM, indicating whether or not they are suffering resource constraints.

Veeam ONE, from Veeam Software, is a great all-around performance tool and is available in both free and commercial versions. In fact, Veeam offers a full commercial version for VMware Certified Professionals for their lab environment.

VMturbo Community Edition is the free version in VMturbo Inc.’s product lineup and is deployed as a virtual appliance with a Web interface. Even for a free tool, it provides valuable capacity analysis information.

Of course, if you do have a production environment with a commercial performance tool, many admins just use the same license in their lab that they use in production.

Having a vSphere lab is crucial to the success of any vSphere virtualization professional. It's used to test (so you don't cause downtime in production), to demonstrate (for customers and fellow IT pros), and to learn (help you pass your VCP or VCAP). However, if performance bottlenecks cause it to run poorly, it isn't useful for much of anything.