10 ways to improve virtual server storage
Virtual server storage operations can have a profound effect on the storage network. Here are 10 ways to help ensure that your storage yields its best I/O for the VMs it supports.
Virtual server storage operations can have a profound effect on the storage network. Here are 10 ways to help ensure that your storage yields its best I/O for the VMs it supports.
Enterprise virtual infrastructure will always use shared storage. It's a fact of life -- if you want to use the advanced features of VMware vSphere and Microsoft Hyper-V, all your hosts will need access to the files that make up your virtual machines (VMs). While the latest editions of VMware VMotion and Microsoft Live Migration don't require shared storage, most features like VMware vSphere High Availability and Microsoft Failover Clustering still do (and likely always will).
For the virtual infrastructure (and your critical applications) to perform well, they must be fed the virtual CPU, virtual memory and virtual disk I/O they hunger for. The monitoring of virtual CPU and memory has gotten easier as hypervisors have made more statistics available and monitoring tools have gotten smarter. However, performance monitoring for virtual infrastructure storage is still, in many cases, more of an art than a science. Certainly there are scientific statistics that are analyzed in storage performance, but the difference is that the storage is external to the hypervisor and physical server and, thus, the hypervisor (and your virtualization performance monitoring tools) doesn't typically have as much insight into what's happening with storage (but that's changing fast).
Nonetheless, there are a number of steps you can take to improve the performance of your virtual server storage. Here are 10 tips that can help you improve your storage systems' performance for your VMs.
1. Know your IOPS
The science of tuning virtual server storage is, in its simplest form, "IOPS in versus IOPS out." The drives in a LUN can provide a certain number of I/Os per second, the path in between the server and storage has a specific throughput, and the applications demand so much. While the simplest question is "Do you have enough or do you need more?," the answer isn't always easy. Many of the tools, technologies and solutions described in the following tips can help you gain more I/O capacity or find ways to reduce the I/O demand (usually by knowing the apps and working with app owners or developers).
In a virtual infrastructure, the best way to understand the typical I/O for VMs and datastores is to use virtualization-specific performance analysis tools. Almost every one of these tools will help you to "know your IOPS." Those tools could be the included VMware esxtop CLI tool, performance graphs inside vCenter (where IOPS are represented as disk.numberRead.summation and disk.numberWrite.summation counters) or commercial tools from third parties. Because vCenter stats must be divided by the sample time to produce IOPS, it can be more work to calculate. Third-party tools (or even VMware vCenter Operations Manager) are more costly but are much easier to use and much more intelligent.
Note that, among other things listed below, LUN configurations and RAID selection can also have a massive impact on I/O performance.
2. Know your applications
While storage and virtualization admins have many tools to help churn out more I/O, they're usually limited in the ways they can reduce the I/O demand. Also, too many IT pros are more than happy to get knee deep in the weeds of data center infrastructure while totally ignoring the applications running inside.
It's tough to be successful at truly understanding storage performance if you don't know what your most I/O-intensive applications are, what VMs they run inside, what server clusters they run on and what storage they use. Are the applications read- or write-intensive? Knowing the answers to these questions will guide you in your storage tuning efforts. By understanding the applications, and working with app owners and app developers, you may even be able to reduce the I/O workload and save the company a small fortune by not having to buy a new storage array or add high-performance drives.
If the application is already virtualized, you can use the tools mentioned above to get to know the typical I/O load over time and whether it's read- or write-intensive. If the app isn't virtualized, you may want to create a proof of concept for virtualizing the app, test the virtualized app with a realistic load and then model the production workload based on those tests. Alternatively, if the production app already uses a SAN, then you could use your SAN vendor's performance tool to measure I/O workload for the physical server before virtualizing.
3. Maximize the storage you have
One way to improve storage performance is to ensure that you're making the most of the storage you already have. You may have features that aren't even turned on, or perhaps you have features that could be enabled for a minimal cost. You could even consider reconfiguring your storage LUNs or RAID groups to gain better performance for the virtual infrastructure. I/O workloads change over time and perhaps the LUN layout and RAID groups that were laid out a couple of years ago no longer apply. By rebalancing the workload across different LUNs or changing the RAID type for a LUN, you may improve performance for the virtual infrastructure as well as physical servers. In many cases, admins tend to look to new "bells and whistles" to improve performance when the source of virtual infrastructure storage performance issues are the same things that typically affect traditional physical servers.
4. VAAI or ODX
VMware's vStorage APIs for Array Integration (VAAI) allows the vSphere hypervisor to offload some storage-related tasks to the storage. For example, instead of cloning a VM through the hypervisor, that can just be done in the storage. VAAI also significantly reduces the time the disk is locked by using hardware-accelerated locking. Make sure your storage is VAAI-capable to ensure that your virtual infrastructure benefits from storage offload.
In the Microsoft Hyper-V world, Offloaded Data Transfer (ODX) is comparable to VAAI. If you're using Hyper-V, you want your storage to be ODX-capable.
To find out if your storage offers VMware VAAI, look up your SAN or NAS in the VMware Compatibility Guide. To find out if your storage is ODX compatible, ask the vendor if their product is Microsoft ODX certified. Over time, more and more vendors will add VAAI and ODX compatibility to their enterprise storage arrays.
5. Limit snapshots
One of the most useful features of virtualization is the ability to snapshot a virtual machine. That point-in-time picture of the VM's memory and virtual disk are useful should an application or OS upgrade go wrong or you need to test configuration changes. The snapshot is also used by virtualization backup and replication applications to capture virtual disk changes without causing downtime. However, too many times snapshots are overused, not only taking up disk space but causing poor performance for backup applications, VMotion and other storage-related functions. Snapshots should only be used temporarily and then deleted.
6. Thick provisioning versus thin provisioning
The ability to create VMs with thin provisioned virtual disks is very useful. I use thin provisioning all the time in my lab environment. However, thin provisioning in the virtualization layer and physical storage array must be carefully planned and managed. Most of the problems with thin provisioning relate to simply running out of storage. However, if you don't have VAAI-enabled arrays, there can be some performance degradation when the thinly provisioned virtual disk needs to be increased in size. In that case, you may want to use thick provisioning (lazyzerothick, to be specific) for VM disks and then use thin provisioning in your storage array.
7. Virtualization-aware storage
Some new storage arrays are billed as "virtualization-aware." That means the storage talks to VMware vCenter, and so the storage knows what VMs are running and the path to their virtual machine disk files (VMDKs) on the SAN/NAS. Both the virtualization admin and the storage admin benefit significantly from this communication. For example, you could find out what VMs are creating the most IOPS or which VMs are experiencing the most storage latency. Some virtualization-aware storage systems can even provide per-VM snapshots and clones. You don't necessarily have to replace your storage array with one that's virtualization-aware, but it's something to consider for future projects and something to ask your current storage vendor about to see what similar features they're planning for future release.
8. Partition alignment
When older operating systems (OSes) are run in VMs without taking partition alignment into account, you can cause performance issues. However, with Windows Vista, Windows 7 and Windows 2008 (or later, such as Windows 8 and Windows 2012), this is no longer an issue. Some virtualization-aware storage can automatically align VMDKs, no matter the OS. But if your company is using older OSes, you should verify that your VMDKs are aligned with your virtual machine file system (VMFS) and SAN LUN.
There are a number of tools available to help you verify partition alignment of pre-Windows Vista and pre-Windows 2008 OSes, including the free UberAlign or the commercial vOptimizer Pro.
9. Storage I/O control
With multiple virtualization hosts sharing the same storage, something needs to ensure the storage isn't monopolized by an I/O-hungry application or that more critical apps aren't starved for I/O. VMware's vSphere Storage I/O Control does this and is enabled with a single checkbox on a VMFS datastore.
10. Storage Distributed Resource Scheduler (SDRS)
What VMware's Distributed Resource Scheduler does for CPU and RAM, SDRS does for storage. SDRS will move virtual machine disks (with no downtime) from one datastore to another (or even from one array to another) should that datastore be experiencing high latency or high disk space utilization. Make sure your storage can speak to vCenter using the VMware vStorage APIs for Storage Awareness (VASA) so vCenter can identify the capabilities of the storage for SDRS to be as successful as possible in preventing storage hiccups.
And if all else fails…
While budgets are tight for everyone, you may need to spend some money and purchase a new storage system to get higher performance for the virtual infrastructure. The virtual infrastructure's I/O workload of 50-plus VMs all hammering on disk LUNs will be very different from the previous 1:1 mapping for a server and a disk LUN. In addition to the better performance your new storage will provide, you may gain additional features such as virtualization-aware storage, hybrid storage, VAAI, VASA and more.
Virtualization is a new mindset in the data center; when VMs are consolidated, storage must be rethought and re-equipped.
About the author:
David Davis is the author of the VMware vSphere and Microsoft Hyper-V video training library from www.TrainSignal.com. He has written extensively, is a vExpert, VCP, VCAP-DCA and CCIE #9369 with more than 18 years of enterprise IT experience.
