VMware vSphere hot-add RAM and hot-plug CPU: Not so 'hot,' but still cool

virtual machines. The benefit of being able to do this is the ability to provide more resources to your machines without bringing servers down to add the additional resources. Simply put, this is additional capacity without downtime.

But there is much more to using hot-add RAM and hot-plug CPU than checking a box or flipping a switch. In the following article, I will cover the 10 things that I think a VMware administrator needs to know about vSphere hot-add RAM and hot-plug CPU.

1. There is a difference between "hot-add" and "hot-plug"
Yes -- there is a difference between these two phrases, but honestly, it's just semantics.

If you add more RAM to a running VM, then you are 'hot-adding' it. If you add another virtual CPU (vCPU) to a running VM then you are 'hot-plugging'. it would be incorrect to say that you used hot-plug RAM or hot-add CPU. If you do, your coworkers will likely know what you are talking about but your terminology would be wrong, at least according to the VMware documentation.

2. Hot-add and hot-plug are not enabled by default
There are 'toggle switches' that must be enabled in the configuration for each virtual machine in order for that VM to support hot-add or hot-plug. Some caveats: Whether or not the switches appear in a VM's configuration depends on the OS that vSphere is told is running inside that VM. In other words, if you configure a DOS VM and go into the configuration, there won't be an option to even enable hot-add/hot plug. However, if you create a new Windows 2008 Server VM, then you will see the proper switches. Secondly, the ability to use hot-add/hot-plug is not enabled by default. So, even if you create a new VM that is compatible with hot-add/hot-plug and power it on, the ability to use those features is disabled by default.

Also, keep in mind that to use hot-add/hot-plug, your virtual machine version must be version 7 or greater. If it isn't, you'll need to upgrade your virtual hardware first.

3. To enable hot-add/hot-plug, the VM must be powered off
If hot-add/hot-plug is not already enabled when a virtual machine is powered on, you must shut down the virtual machine before you can enable it.

4. There is a difference between hot-add and 'hot-use'
You may have the ability to hot-add RAM to a running VM, but that doesn't mean that the guest OS is going to recognize that RAM and allow you to use it. Depending on the guest OS, you may or may not have to reboot it to get it to recognize the new hardware.

The same goes for hot-plug CPU. You may be able to hot-plug a new CPU but you may have to reboot your guest OS in order for it to recognize the new CPU.

5. The success of hot-add/hot-plug is very dependent on the guest VM's OS
You can't use hot-add/hot-plug with just any guest OS. You need to be very careful here! VExpert and noted virtualization blogger Jason Boche tested hot-add/hot-plug with various CPUs and posted the results. Even better, he created a nice table of those results.

Click to enlarge. Full article and table at: vSphere Memory/CPU Hotplug (hot add) .

One of the most important things to note in the table is the small asterisk next to some of the operating systems. That asterisk means that you must reboot that guest OS so that it will recognize the new RAM or new CPU (which is contradictory to the name of the function, as it's called "hot-add" and "hot-plug").

Of course, these issues aren't VMware's fault. It is the job of the OS manufacturers to make their OSes compatible.

In addition to the guest OSes listed on the table above, I can tell you that I have tested Windows 2008 64-bit Datacenter Edition and it is compatible with vSphere hot-add/hot-plug. Rebooting is not required to add RAM or CPU, and the guest OS will recognize the additional resources without additional reconfiguration. Also, Windows 7 64-bit Ultimate and Enterprise have been rumored to be compatible with hot-add/hot-plug without having to reboot. However, I encourage you to test those versions.

You should also verify that your applications can actually use the new CPU that has been added. For more information on that, see  Hot Add CPU – vSphere.

6. Enabling hot-add and hot-plug
You have to enable hot-add/hot-plug just after you create a new VM, but before you power it on and install the OS. If you don't, you'll have to shut the VM down.

To enable these functions, go to the VM in the vSphere client, click on Edit Settings, click on the Options tab. Under Advanced, click on Memory/CPU Hotplug option and choose to enable memory hot-add and CPU hot-add for this virtual machine.

Click to enlarge.

7. Hot-add/hot-plug is not compatible with VMware Fault Tolerance
That's right, you heard me (err, read me) correctly: hot-add/hot-plug is not compatible with Fault Tolerance (FT). Honestly, I wish they worked together because both FT and hot-add/hot-plug are amazing features of vSphere.

8. Whether or not you have the ability to hot-add/hot-plug is based on your version of VMware vSphere
Hot-add and hot-plug are not available in every version of VMware vSphere. In fact, you only have access to the hot-add/hot-plug features in the Advanced, Enterprise, and Enterprise Plus versions of vSphere. For more information, see VMware Knowledgebase article Comparison of product offerings for vSphere 4.0 and VMware Infrastructure 3.X.

9. Linux supports hot-add but not hot-plug
As of today, I haven't found anyone that has found a Linux OS that supports hot-plug of additional vCPUs. I fully expect that Red Hat Enterprise Linux (RHEL) or Suse Linux Enterprise Server (SLES) will add support soon.

While I haven't tested it myself yet, I am told that most versions of Linux with a kernel version of 2.6.14 or greater support hot-add of RAM. Here is an example where they describe how they took advantage of hot-add RAM in CentOS – vSphere ESX4: Hot Add Memory for Linux Guests.

10. You can hot-add other devices
Yes, hot-add RAM and hot-plug CPU is new but you can hot-add other types of devices. For example, you can add a second virtual disk to a running VM, then go and partition it and format it in the guest OS with no downtime. You can add an additional virtual network interface card (NIC). You can even expand the root drive of a Windows Server while the guest VM is running, as long as it is a dynamic disk.

In summary, hot-add and hot-plug are (forgive me) "hot" new features that give VMware admins the power to add additional capacity to critical production systems with no downtime to the end users. However, the ability of the guest operating systems to support this new technology without having to reboot the guest OS is still missing and would be a greatly welcomed enhancement.

David Davis is the director of infrastructure at TrainSignal.com. He has a number of certifications including vExpert, VCP, CCIE #9369 and CISSP. Davis has also authored hundreds of articles and six different video training courses at Train Signal with his most popular course being VMware ESX Server. His personal website is VMwareVideos.com. You can follow Davis on Twitter or connect with Davis on LinkedIn.