Virtualization 101: Best practices for securing virtual machines

Virtual machine discovery and inventory

A first critical step in properly securing a virtual infrastructure is ascertaining where virtual machines are located and how an accurate inventory can be maintained. In many organziations, system inventories are out of date; in fact, many are kept in spreadsheets with manual input from systems and network administration teams.

Due to the dynamic nature of virtual environments, a common scenario dubbed virtual sprawl can easily occur, where virtual machines are created and used for a period of time, but never noted in a formal systems inventory. Many of these virtual machines may be used for testing or short-term purposes, and remain active long after they've served their initial purpose. Unfortunately, with little lifecycle maintenance, these systems can easily be missed during patching cycles, and may expose your organization unnecessarily.

There are many ways to maintain an accurate virtual machine inventory via discovery and systems management tools. For many virtualization deployments, inventory can be maintained by using built-in tools within virtualization platforms, such as the inventory category within VMware vSphere's vCenter management console, or Microsoft's virtualization management tools such as Systems Center Virtual Machine Manager. Other tools can be leveraged, as well, such as VMware Lifecycle Manager, which offers more robust system lifecycle management and provisioning, or endpoint security and configuration tools that rely on installed agents within virtual machines, such as Symantec Altiris and similar products. Finally, assessing the known inventory on a hypervisor platform such as VMware ESX or ESXi can be accomplished with various scripting tools.

In addition to these tools, several other discovery options should be considered. First, because most virtualization deployments rely heavily on centralized storage, any available storage management tools can be leveraged for VM file inventory maintenance. As most, if not all, virtual machine disk and configuration files will be stored on a storage area network (SAN) or network attached storage (NAS), any inventory tools from storage vendors should be used to the fullest extent possible. Examples of these include EMC Ionix ControlCenter and NetApp OnCommand products. Second, verifying running virtual machines from a network perspective can be done using well known network scanners such as Nmap and others--all virtualization vendors have a defined set of organizationally unique identifiers (OUIs) in place for the first three hexadecimal values of a virtual system's MAC address. By scanning local subnets and capturing MAC addresses and comparing them to these OUIs, security teams can correlate this data with other inventory information.

Virtualization change and configuration management

The second major area to consider in properly securing a virtual environment is operations management, namely change and configuration management. At the 2008 Burton Catalyst conference, Alessandro Perilli, founder of virtualization.info, stated that "[t]he weakest part of the security defense we have in our infrastructure is related to the way we manage our operational framework."

Unfortunately, little has changed since 2008. Integrating virtualization platforms, management infrastructure, network components and virtual machines into existing change and configuration management policies and procedures is critical to ensure long-term stability and security of the entire infrastructure, particularly as the use of virtualization increases.

Configuration management is primarily focused on two elements: security hardening and patching. From a security hardening perspective, numerous sources of guidance exist to help systems and security administrators adequately lock down their virtualization components.

For hypervisor platforms (for example, VMware ESX, Microsoft Hyper-V, and Citrix XenServer), most major vendors have guidance freely available. The latest version of VMware's vSphere Hardening Guide includes guidance on configuring virtual machine configuration files, hypervisor hosts, virtual networks, and management components, with flexible options for different levels of security criticality.

Microsoft's Hyper-V Security Guide outlines several important configuration practices that should be considered for any Hyper-V implementation, such as running Hyper-V on 2008 Server Core, and selecting specific server roles, implementing Authorization Manager for more granular roles and privileges, and hardening Windows virtual machines. In addition, the Center for Internet Security (CIS) and the Defense Information Systems Agency (DISA) have free configuration guides available for download at their respective sites. These guides should be viewed as a starting point for proper security hardening, since most organizations will have numerous modifications and concessions required for their own operating environments.

Patching virtualization infrastructure is the second critical configuration task that should be performed regularly. The first option for many security and operations teams will be to investigate their existing patch management product(s) to see whether they support virtualization products and platforms. In most cases, the hypervisor hosts will need to be patched with specialized tools, such as VMware Update Manager. The virtual machines can almost always be patched with existing tools, although specific scheduling and testing regimens may be called for. There are two primary differences to consider when patching virtual machine operating systems. First, patching will need to be carefully scheduled so as not to overload the shared pool of physical resources on a single platform, such as RAM, CPU, etc. The second consideration relates to offline, or "dormant" VMs -- these will need to powered on in order to patch in most cases. Be sure that your patch management tools have been tested to work with whatever type of virtual machines you're running (Xen, VMware, etc.).

Change management is another key element of secure and resilient operations for virtualization. Virtual machines can be created and made available within minutes, versus traditional servers and applications that need to be installed on hardware and installed in a data center. For this reason, it's imperative that new change management ticket categories are created for producing, modifying, and deleting virtual infrastructure or virtual machine components, and virtualization teams should be included in all change management review meetings and discussions. Provisioning, patching, updating and decommissioning virtual machines should be done exactly the same way as their physical counterparts from a process and policy standpoint, and this needs to be reinforced from the highest levels of IT management.

Security architecture for virtual networks

There are many architecture options security and network teams will need to consider for virtual network environments. First, virtual switches are different in many ways from physical switches. Many more switch ports can be provisioned on a single virtual switch than a physical one. Also, default virtual switches from virtualization vendors cannot be cascaded, or connected to each other, inside the virtual environment. For this reason, planning the number and types of virtual switches that need to be connected to physical NICs is critical, because the number of physical NICs in a system is limited.

This also means that virtual switches are isolated from each other by default, and most also support the use of virtual LANs (VLANs) for additional Layer 2 segmentation between specific groups of ports on the virtual switch. Some virtual switches also have built-in security policy settings that can be configured. For example, VMware's default virtual switch can be placed into promiscuous mode for monitoring, and can also have rudimentary MAC address filtering enabled to prevent MAC spoofing attacks.

However, the default virtual switches from platform providers leave much to be desired. True SPAN or mirror ports cannot be created for dedicated traffic mirroring, extensive port-level security is not available (locking down one port to one MAC address, for example), and management capabilities are very limited. Cisco has created a virtual switch, the Nexus 1000v, which can be imported into virtual environments and offers the same features and functionality as a traditional physical Cisco switch, complete with command-line IOS management. For Citrix, KVM, and VirtualBox environments, the Open vSwitch virtual switch is an open-source alternative that provides similar functionality to Cisco's offering.

Regardless of the virtual switches used, security teams will want to ensure that redundancy and security are built into the virtual network design. Several different traffic segments are typically associated with virtualization platforms. The first is simply the virtual machine production traffic, consisting of virtualized operating systems and applications. This traffic should be on separate virtual switches, with at least two physical NICs for redundancy. The next traffic type is storage traffic and specialized virtualization traffic, often including virtual machine migration that may occur in cleartext. Since this is very sensitive data, this segment should be on distinct virtual switches when possible, with multiple dedicated physical NICs for redundancy, as well. Finally, a third segment should be in place for management traffic, usually consisting of protocols like SSH and SSL-based management console interaction. Like the other two segments, separate virtual switches and redundant physical NICs should be used.

A core tenet of virtualization is the ability to have multiple virtual machines and networks on a single physical platform. By default, virtual machine traffic on different virtual switches is separate, unless both virtual switches connect to the same physical network outside the hypervisor platform. However, all traffic is handled by the hypervisor, and a potential compromise to the hypervisor could allow traffic to be exposed at a single point. For this reason, it is recommended that data of different sensitivity or classification levels be kept on separate physical hypervisor platforms as an added measure of segregation.

Hypervisor security and management

One of the most commonly overlooked elements of virtualization security is proper management and administration of hypervisor platforms and related components. In many cases, a single systems administration team is charged with designing and managing all aspects of the virtualization infrastructure, but this violates the security best practices of separation of duties and least privilege.

To properly maintain these principles, specific roles and groups should be created within the virtualization management console or similar third-party application that allows network teams to manage virtual networks, specific administration teams or development teams to manage particular virtual machines, and a core virtualization team (or other administration team) to manage the general virtualization platform configuration. Additional roles may be needed for auditors and security teams, depending on the scenario. Only the specific privileges needed for these roles should be assigned--in other words, networking teams have no need to manage virtual disk images, auditors should be granted "read only" access, etc.

Management platforms should also be secured properly. These systems should be considered high value, as they grant full access to the configuration of hypervisor platforms, virtual machines, virtual networks and storage components in use. Many management applications are installed on Microsoft Windows operating systems, and keeping these systems patched and locked down appropriately is critical to the overall security of the entire virtual environment. Regardless of OS, make sure to keep the management systems on a separate, carefully restricted network segment that is only accessible to approved administration teams, and institute sound log management practices for all access to the systems, failed logins, error messages, and other events dictated by security policies and compliance requirements.

Tweaking security technologies for VMs

There are many additional security technologies and processes that are likely affected by virtualization. For example, antimalware agents running on virtual machines must be configured to exclude certain virtual disk or configuration files (to prevent corruption), and file system scans must be scheduled very carefully, to avoid multiple virtual machines using shared hardware resources simultaneously, potentially leading to a local denial-of-service or other undesirable consequences.

Intrusion detection systems and firewalls may not have granular visibility into the virtual environment to enforce access controls or detect anomalous or malicious traffic. For this reason, many security product vendors have created virtual appliances for these devices, allowing internal virtual switch traffic to be monitored and controlled much like that in traditional physical networks.

McAfee, Symantec, Sourcefire, HP TippingPoint, and many other vendors have virtual offerings for intrusion detection and prevention systems. A number of companies offer products specific to virtual network access control and traffic analysis, such as Altor Networks (now Juniper), Reflex Systems, and HyTrust.

Virtual appliances for mail and network antimalware gateways are available, and VMware has a number of security products available in their vShield line, including traditional and application-centric access control systems, as well as antimalware capabilities. Open-source offerings such as the Snort and Shadow IDS engines, as well as the host-based OSSEC IDS can be downloaded as virtual appliances or installed into virtual machines, too. In general, most security professionals feel that virtualized security tools should be used to augment existing security technology instead of replacing it, but these new tools will most certainly be more readily adopted over time.

Although many IT teams may make the argument that virtualization simplifies the infrastructure, the opposite may be true for security professionals. The use of virtualization technology adds additional layers of complexity and interaction between applications, operating systems, hypervisor engines and network components. New management systems, storage requirements and data protection scenarios, such as automated migration of virtual machines from one system to another, make security and controls maintenance challenging as virtualization continues to grow. Many best practices are still applicable, however, and by diligently applying security to design, discovery, and configuration processes, it's possible to create a secure virtual infrastructure today.

Dave Shackleford is a founder and principal consultant with Voodoo Security and also a certified SANS instructor.