Advanced ESXi troubleshooting with Tech Support Mode

After Tech Support Mode is enabled, entering VMware troubleshooting commands in the Linux command line will provide a better understanding of what’s ailing your ESXi host -- especially when the vSphere Client can’t do the job.

The vSphere Client is the primary administration and troubleshooting tool for VMware admins. But there are times when the vSphere Client can’t connect to an ESXi host or when server hardware is causing an issue. In these situations, you can enable Tech Support Mode and issue VMware troubleshooting commands on an ESXi command line to diagnose and resolve problems.

Once Tech Support Mode is enabled, you can use commands found on every Linux distribution (e.g., /bin/ls, which list files, and /bin/cp, which copies files). There are also a few VMware-specific commands that are very useful for troubleshooting hosts:

• vmkvsitools, which is a master command that can retrieve detailed information about the host and the ESXi processes, depending on the subcommands that you include;
• vim-cmd, which is another master command that allows you to perform several VMware troubleshooting and administration tasks, such as powering on virtual machines and installing VMware Tools; and
• esxtop, which lists all the of the system processes running on the ESXi host.

Using vmkvsitools
By adding arguments to the vmkvsitools master command, you can drill down for specific data about the ESXi host. To get an overview of available commands, run vmkvsitools from inside the /bin directory. 

Figure 1
Run the vmkvsitools command to find out which commands are available. (Click image for an enlarged view.)

As a shortcut, you don’t need to add vmkvsitools when running a subcommand. For instance, to find out the amount of available disk space in the host’s file systems, you can run either vmkvsitools vdf or just vdf.

Figure 2
The vdf command retrieves the available disk space. (Click image for an enlarged view.)

Another useful command in vmkvsitools is lspci, which lists the devices found on the PCI bus. This command is useful for troubleshooting because it details the hardware that’s installed in the server. Based on that information, you can find out if VMware supports that hardware.

Figure 3
Use lspci to find out which hardware devices are found on server. (Click image for an enlarged view.)

The power and complexity of vim-cmd
You can use the vim-cmd commands independently or in scripts to manage and troubleshoot virtual machines from the command line. As such, the commands provide management options that are normally found in the vSphere Client, but you have the advantage of using them directly on your server that’s has Tech Support Mode activated.

Like vmkvsitools, vim-cmd is a master command that organizes its subcommands in different subdirectories. To see the vim-cmd subdirectories, run vim-cmd without any argumentsUse lspci to find out which hardware devices are found on server.

The power and complexity of vim-cmd
You can use the vim-cmd commands independently or in scripts to manage and troubleshoot virtual machines from the command line. As such, the commands provide management options that are normally found in the vSphere Client, but you have the advantage of using them directly on your server that’s has Tech Support Mode activated.

Like vmkvsitools, vim-cmd is a master command that organizes its subcommands in different subdirectories. To see the vim-cmd subdirectories, run vim-cmd without any arguments.

Add a subdirectory at the end of the vim-cmd to see all the commands in that subdirectory (e.g., enter vim-cmd vmsvc for a list of all the commands for virtual machine management). To run a specific command, add it at the end of the vim-com and subdirectory (e.g., type vim-cmd vmsvc/getallvms for the configuration details of every VM on the host).

Figure 4
The vim-cmd master command allows for a broad range of management and troubleshooting options. (Click image for an enlarged view.)

The difficulty with vim-cmd commands is that there isn’t much information on the Web about them. Remember, these commands were meant for VMware technicians, and Tech Support Mode wasn’t supported for the general public until ESXi 4.1. Nevertheless, But a couple of useful vim-cim useful commands include vmsvc/get.environment (which lists statistics, such as the number of CPUs and CPU cores as well as data store information) and vmsvc/snapshots (which creates VM snapshots). 

Using the esxtop command for VMware troubleshooting
Finally, esxtop is a useful VMware troubleshooting command that is available with Tech Support Mode enabled. It’s similar to the top command in a Linux and lists of all system processes for a host and the its virtual machines, including the CPU, memory and disk utilizations. 

These numbers are expressed as a total percentage of system time, where every CPU core on your host equals 100%. Thus, the processes running on So a four-core CPU will can equal 400%, which is why a process may exceed 100%. 

Esxtop is useful if you experience sluggishness on your host as it may help you find which process is responsible. Armed with that information, you can develop an effective VMware troubleshooting plan. 

Figure 5
Esxtop can help you identifying processes that heavily affect system performance. (Click image for an enlarged view.)