Host-based intrusion prevention addresses server, desktop security

HIPS is used for everything from traditional signature-based antivirus/antispyware and host firewalls to behavior analysis.

Patty Long has a thousand reasons to deploy host-based intrusion prevention: 1,000 DMZ servers, database servers...

and application servers.

"We were looking across the organization and trying to touch every single application, every server where data--participant data, sponsor data, plan data--would transverse our environment, to make sure we have every point where it's at rest or could pass through covered," says Long, who is with ING's Secu- rity and Risk Management Department. She deployed Third Brigade Deep Security at CitiStreet, which was acquired by ING Group this year (now part of ING Wealth Management).

Most people still think in terms of the original host-based intrusion prevention systems (HIPS) technologies, which monitored OS system calls for anomalous behavior. The best known were Okena's StormWatch, which evolved into Cisco Systems' Cisco Security Agent (CSA), and Entercept Security Technologies, whose products became McAfee Host Intrusion Prevention.

Today, HIPS encompasses many technologies to protect servers and/or desktops and laptops. Many experts and analysts use it as an umbrella for everything from traditional signature-based antivirus/antispyware and host firewalls to behavior analysis.

Long is certainly not unique, but while host-based intrusion prevention systems are moving into the mainstream of information security, they are not generally well understood by most organizations.

"It's still a new frontier for them. We have to explain what it is, what it's doing," says Ed Skoudis, co-founder and senior security consultant of Intel-guardians. "What a lot of them remember is Okena and Entercept; that's a piece of it, but today it goes way beyond that."

Faced with signature-defying malware and regulatory demands for tighter controls over corporate systems and data, traditional AV vendors have been scrambling to add HIPS, both standalone and as part of endpoint suites. And, HIPS vendors are bundling AV capabilities, generally through OEM deals.

So, you have vulnerable servers open to all HTTP traffic and roaming, hard-to-control laptops, and a smorgasbord of host protection options.

When Information Security first wrote about HIPS in 2002, signature-based AV was still king. Most attacks were what one security expert recently referred to as "Internet hooliganism" rather than profit driven. Antispyware was a consumer-oriented cottage industry.

Today, signatures ain't enough.

"The bad guys are completely overwhelming signature-based antivirus," says Skoudis. They're appropriate for the threat model we faced five to 10 years ago. Some vendors brag that they're getting 90 percent detection. If you give the vendors the benefit of the doubt, that's awful. If my machine has 100 exposures a week--that's on the low side--it still means I'm infected 10 times."

So, we have a wide range of protection technologies, offered in different combinations by numerous vendors--but no silver bullet. To make intelligent buying decisions, it's important to understand the ways these technologies detect attacks. Each has its strengths, limits and, in some cases, risks.


Everything that can run on your server or PC is either a known bad app, a known good app or--and this is where things get really dicey--unknown. Traditional signature AV pretty much does the trick against the stuff we know is bad. Known good is an authorized application. (Or, if it's benign but not part of your company's standard build, you can ban or limit its use by policy.) The unknown--the zero-day attacks, the multitude of malware variants--is where criminals live large.

The early HIPS models waited for the code to start executing, attempted to determine if it was making legitimate system calls, and blocked them if it was not.

Today, you can attempt to detect an intrusion when it attempts to penetrate a system, when it's on the system but hasn't executed, or when it executes. This applies to remote network-based attacks or ones that require a Web surfer's download or clicking on an email link or attachment.

So, HIPS technologies can be generally categorized by what they look for, how they look for it, and at what stage they attempt to detect it. Neil MacDonald, VP and Gartner fellow, draws this up into a nice matrix of nine styles of protection. He cross-references the three states of our knowledge about the code (known bad, known good, unknown) and three stages at which they can be detected--network-based, when the code is attempting to penetrate the system; application-based, when it is on the system; and behavioral, when it executes.

MacDonald calls the latter "the last line of defense," because the code is already executing. Some solutions mitigate the risk by allowing the code to run in a sandbox, so its behavior can be analyzed without putting the system at risk, or using an emulator to model what it would do if allowed to run unfettered.

Some sort of behavior-based detection is key to assessing unknown code. One technique uses behavioral "signatures"--it identifies common bad behavior, such as misuse of memory or modifying OS files. Instead of specific malware signatures, it looks for the common techniques of a given class of exploits. Or, it may look for attack behavior designed to exploit known vulnerabilities. Others monitor behavior and create a baseline of normal activity.

In addition to the inherent risk of allowing code to execute, behavioral techniques tend to produce false positives and eat up more system resources.

In the worst cases, they can destabilize and crash systems. MacDonald refers to this as a potential denial of service for the underlying system.

"The downside of behavior-based techniques is that while they can detect unknown attacks, which signature antivirus can't, their tight coupling with system API calls makes them brittle," says Josh Corman, principal security analyst with IBM, which sells Proventia Desktop Endpoint Security and Proventia Server Intrusion Prevention System.

This is especially true for servers, Corman says, "where availability is king."

"It depends on environment," says Intelguardians' Skoudis. "If your environment is very constrained, if you can be fairly draconian, if you don't need your users to run all kinds of crazy apps, whitelisting solutions are very nice.

"For environments that need to be more flexible, behavior-based is a good way to go--in addition to signatures, of course."

None of these has to be an either-or decision. The key is choosing the right combination of technologies for your environment--for the least money. AV vendors are packaging multiple technologies in addition to signature detection, either through development, acquisition or OEM; meanwhile, some HIPS vendors are bundling signature-based AV.

"Incumbent AV providers increasingly understand and are aware that signature-based protection is not sufficient," says MacDonald, "so they've been supplementing their AV with other styles of protection. The line between point solutions and what, say, Symantec or McAfee can do is pretty blurry."


Logical choice

Virtualization is fertile ground for HIPS.

Virtualization may turn out to be the killer technology for host-based intrusion prevention. The business drivers for virtualization are irresistible--consolidated data centers and storage, huge energy savings--but security, typically, is playing catch-up.

Network firewall and intrusion prevention vendors are moving to virtualization solutions-- Check Point Software Technologies announced a virtual appliance firewall, and Sourcefire and IBM are releasing virtual IPS appliances.

In an environment when virtual machines are going up and down, even moving dynamically from physical server to physical server, the argument for host-based protection that carries security policy with it becomes pretty convincing.

"The nature of the architecture does promote a host-based control that moves with the machine," says George McTaggart, VP of marketing for Third Brigade. "There's a lack of visibility where you had network chokepoints before. Management is about putting the right rules in the right place for the right amount of time."

The case may get even stronger as VMware's VMsafe initiative (and similar technologies by virtualization competitors) will, over time, allow security vendors to integrate tightly with the host platform itself. The promise for HIPS in that case is one stance on the physical machine protecting both the hypervisor and its clients. Of course, the same argument can be made for next-gen virtual network IPS appliances.

For now, best practice is to put HIPS on each virtual server. Tread carefully about putting HIPS on the host itself.

"Deploying on the host could impact performance and cause all sorts of problems," says Ed Skoudis, co-founder and senior security consultant of Intelguardians. "Keep an eye on what's going on with VMsafe, but that's in the future; it's not ready for prime time yet."


Server or desktop

HIPS has historically been a server-centric solution. The technologies are often identical, but the use cases can be very different. Servers are generally perceived as your most critical exposure--they run your production apps, house your databases and sensitive files, etc.

They're also a friendlier environment for HIPS. It's a lot easier to detect what should not be running on machines whose purposes are clearly defined and run far fewer applications than a typical desktop or laptop. They tend to be locked down. Enterprises often have tight change control policies and processes for servers; for desktops, to be kind, not so much.

"Host intrusion prevention technologies are still more widely adopted on servers than on desktops," says MacDonald. "They're more mature on servers.

"There's a variety of reasons for that; the most important is that as compared to desktops, servers lead really boring lives."

The desktop is a lot more challenging. Pesky users are browsing the Web downloading God knows what, opening email, plugging in USB sticks, and so on.

"You don't have the ability to introduce arbitrary code on servers," says MacDonald, "so HIPS becomes a preferred way to protect some servers, especially embedded systems, SCADA systems."

Considerations for choosing server technologies include performance and stability. A crashed laptop is a nuisance; a crashed server is a calamity.

For example, whitelisting makes sense for a highly controlled server. Say, it's running IIS, maybe a database, perhaps a custom application; then look at all the stuff running on that laptop, from your standard MS Office apps, to Adobe reader, Java Runtime, specialized business apps and whatever the user adds, from Skype to World of Warcraft.

It's no accident that vendors only started adding HIPS into their endpoint security products in the last couple of years. An Information Security review of leading endpoint security suites last year showed they have a long way to go in terms of integration, stability and central management.

So, you should be focusing your HIPS plans on servers, right? Not necessarily.

Skoudis, who conducted the endpoint security review, says your desktops and laptops should be your first priority, largely because of all the things that make them difficult: exposure to foreign code, mobility, out-of-control users, etc. Servers are often hardened and sit behind firewalls. And, clients are a gateway to those servers.

"In pen tests, we're often getting access to servers by exploiting the client and riding in on it," Skoudis says. "We're using the client as an attack vector against servers--again and again and again."


HIPS Vendors

The lines are blurring as traditional AV vendors offer and incorporate HIPS technologies, and HIPS vendors include signature-based AV options. This is a representative list of these diverse vendors.

Company Product Description


Bit9 Parity Application whitelisting for data protection, antimalware. Audits and activity monitoring.


CA Host-Based Intrusion Prevention System Behavior-based malware detection and firewall for clients and servers.


Check Point Software Technologies Check Point Endpoint Security Behaviorial/heuristic malware detection, firewall, plus optional signature detection from Kaspersky. (Includes data security acquired with Pointsec.)


Cisco Systems Cisco Security Agent (CSA) Behavior-based detection for clients, servers and embedded platforms. Optional ClamAV signature detection.


eEye Digital Blink Variety of non-signature detection methods, firewall.


F-Secure Client Security DeepGuard behavior monitoring, heuristics and sandboxing, plus signature AV, firewall.


IBM ISS Proventia Server Intrusion Prevention System, Proventia Desktop Security Variety of non-signature techniques, plus firewall and signature-based detection from BitDefender on desktop product.


LANDesk Host Intrusion Prevention System Whitelisting and behavioral analysis.


Lumension Security Sanctuary Application Control Policy-based endpoint application control.


McAfee Host Intrusion Prevention Behavioral and signature-based IPS rules, firewall and application control. Also available with Total Protection for Endpoint.


Sana Security PrimaryResponse SafeConnect Behavioral analysis attack detection.


Sophos Endpoint and Security and Control Behavior-based HIPS technology, plus signature-based detection.


Symantec Endpoint Protection Network and application behavior analysis, signature-based detection, firewall, application control.


Third Brigade Deep Security Analyzes traffic for protocol deviation, attack behavior and policy violations.


Trend Micro OfficeScan Client/Server Edition OEMs Third Brigade to complement signature-based detection, firewall.

Extra Value

Stopping attacks is not all that HIPS is about. There are other security and even operational advantages.

The one that comes up most often is relief from the Patch Tuesday syndrome, the pressure to test and deploy security patches quickly. It's an enormous drain on resources; HIPS technologies that detect attack patterns against particular vulnerabilities are of great value here.

"Once you have your endpoint protected," says Robert Eastwood, VP for operational risk at Delaware-based WSFS Bank, a Cisco Systems CSA customer, "it gives you enough time to test a patch in your test environment, your development environment, before you release it to production machines."

That doesn't mean you don't have to patch, but the idea is you buy time to test and deploy.

"You can reduce the frequency of patching, switch, say, from every month to every other month," says Ted Doty, product line manager for CSA. "One hosting provider has an SLA of no more than five minutes of downtime per customer each quarter. Now he's on a twice-a-year patch cycle."

Intelguardians' Skoudis has a more skeptical take. He says it's OK to prioritize systems that aren't protected by HIPS, but push hard to get all systems patched quickly.

"I get real nervous about it," he says. "It can help you focus patching over the very short term--by that I mean a day. If you've got the resources, patch everywhere."

Because it sits on the endpoint, HIPS is also a rich source of information about what's going on in your network. It's producing logs that can tell you who or what accessed or attempted to access a server, who used a USB stick to copy information or changed a system configuration. These logs can be analyzed on their own or, better, fed into a SIEM tool to correlate information from other sources and generate alerts.

"When I took over the security program at CitiStreet, we had a lot of data, but we didn't have a lot of information," says ING's Long. "Now, we have SIM, which gathers our firewall logs, network IPS, host IPS, some server logs, application logs. We have data from multiple sources to produce effective information, so we can see what's happening in our environment and react where required."

And, if HIPS is installed across your servers and/ or desktops, you can leverage it to manage security policy.

Contemporary Windows and *nix OSes are built tougher.

If operating systems were bulletproof, enterprises wouldn't need to consider third-party host-based intrusion prevention tools to protect their servers and desktops. They're not, but their native security capabilities toughen them substantially. In particular, ubiquitous Windows OSes--everyone's favorite whipping boys--have added Kevlar vests with more protection mechanisms against attack

Windows Vista, and, more recently, Server 2008, feature several new defenses, including:

Windows Services Hardening. This reduces the attack surface by preventing malware from exploiting system services. Applying the principle of least privileges, Vista and Server 2008 restrict privileges to those specified for it and run under privileged accounts.

Application Isolation. Mandatory Integrity Control sets integrity levels for processes, so that applications cannot use system level processes. This prevents malicious applications from subverting system processes.

Address Space Layout Randomization. Earlier versions of Windows used hard-coded memory addresses, which were easy for malicious code to exploit. ASLR randomly arranges the positions of key data areas each time the system boots, making it look different to malware each time and thwart automated attacks.

Across OSes, Sun Microsystems Solaris 10, many Linux distros and Windows OSes, starting with Windows XP SP2, support hardware-based security, known as NX ("no execute bit") for AMD chips and XD ("execute disable") on Intel. Microsoft calls this Data Execution Protection. The one-bit NX flag indicates whether a page loaded into memory is data, and therefore not executable. This prevents malicious code from executing in spaces reserved for data. Buffer overflows are a prime example.

Solaris 10 employs what it calls containers, which use virtualization to deploy a single application to a VM, say, a Web server, to isolate it and use fine-grained access controls on a per-container basis. So, consider the risks associated with a Web server, as part of a complex system, which listens on port 80 for HTTP and requires root privileges. A Web server running in a container has its own port 80 and root user, isolating it and greatly reducing risk.


"It's a Swiss army knife for any kind of policy you want to deploy through an endpoint," says WSFS Bank's Eastwood. "We can push a policy to prevent the use of thumb drives or CD burners to copy information. We can build network access control rules. We can monitor activity to see who is trying to go against policy, or if you are effectively enforcing it."

That also translates into useful audit and reporting capabilities. Depending on your HIPS product's sophistication, you can generate direct reporting and audit information or push the logs into another tool for those purposes. Eastwood says he's leveraged CSA's console for SOX and GLBA reporting, for example. Long has a similar experience.

"We are able to prove to external parties that specific controls are in place at the server level and at the file level," she says.

In fact, HIPS technology even overlaps into change management. While not as robust as a dedicated change management tool, depending on the products, it may be "good enough" change management when you're getting it as an added value.

The case for HIPS

Host-based intrusion prevention is compelling, but is it essential? Security spending is already taking a larger and larger chunk of the IT budget, and the economic outlook makes new security initiatives a very tough sell. You already have AV and network firewalls, vulnerability scanners and possibly network IPS. Maybe your company is implementing laptop encryption and looking into NAC and data loss prevention. Room for one more?

"HIPS is very hard to cost justify," says Intel-guardians' Skoudis. "Your best bet is to have a vendor who has HIPS integrated into other products you already have. Most organizations don't know where they are infected already. If they spend a little more money and are infected less, they won't notice, unless something really big and awful happens."

You can make a case, although it can be hard to quantify. For example, you can save resources if you deploy HIPS to throttle back your patching program. Figure out the man hours saved if you patch every other month instead of every month. It depends on your appetite for risk versus the potential savings. Help desk costs going through the roof? HIPS can save time and money spent repairing or re-imaging systems. WSFS Bank's Eastwood says that's a key value.

"I would say so," he says. "I've seen situations where viruses have run amok on machines. The cleanup can severely impact your business, and, as with most companies, IT resources are limited."

It's easier to make the case for selective deployment, such as on critical servers, where your risk is greatest. If management won't buy the argument for general deployment, identify the critical servers where the business risk is greatest, where the extra layer of protection is tough to say no to.

"There's not going to be the risk-reward, the cost benefit on every server," says Gartner's MacDonald. "Every server is different--the usage scenario is different, the data they host is different, their exposure to threats is different. For some servers, local HIPS makes sense, absolutely."

A better bet, as with many security technologies, is riding in on the regulatory compliance wave. For example, PCI requires some form of intrusion prevention, and HIPS has demonstrable data protection capabilities. Some would even argue that HIPS on Internet-facing servers would fulfill the Web application firewall requirement, though you might want to talk to your QSA before you go down that road.

There's no question that the ability to monitor and report on everything that happens on a HIPS-protected system will help meet your compliance obligations. Some will maintain that the change/ configuration management capabilities make it an attractive multipurpose buy.

"Many of our customers purchased HIPS out of their compliance budget," says IBM's Corman, "for monitoring to pass their audits--file integrity, log/user monitoring. They're getting some security with their compliance budget."

Look to get the most bang for your buck. Decide which HIPS product may or may not have the combination of security technologies, management capabilities and reporting/auditing features that meet your requirements. Increasingly, HIPS and AV vendors alike are trying to pack more in their offerings to make them more cost-effective. For example, AV giants Symantec and McAfee pack multiple protection technologies into their endpoint security products. Trend Micro OEM'd Third Brigade's HIPS to complement its AV package, while IBM has integrated BitDefender's antivirus/antispyware into its HIPS products. And so on.

"I would look at what my existing AV provider has for me and see if that makes sense," says MacDonald, "and compare that to capabilities available in a point solution. Maybe what you are looking for is there and maybe at no additional cost. Larger vendors have more pieces, more styles of protection, and I can pick and choose what makes sense for a server from this palette.

"I shouldn't have a palette of vendors to have a palette of protection styles."

Dig Deeper on Network intrusion detection and prevention (IDS-IPS)