The enterprise's absolute reliance on its network to run its business puts the onus on IT to ensure the availability, reliability and security of that infrastructure. But defending the network against what is an increasingly virulent and sophisticated threat environment can be an extreme challenge. IT has a wealth of tools to use in this fight, including those that capture volumes of data that can point to any number of potential threats. However, huge volumes of data can completely overwhelm an IT staff, making it difficult to discern the real threats from a harmless anomaly. That's where advanced machine learning can help.
The Ponemon Institute estimated, in total, security analysts waste 21,000 hours a year researching false positives that lead them nowhere. These are hours that would be far better used thwarting actual attacks. However, manually trying to distinguish between actual threats and unusual patterns when so much information exists can be nearly impossible. For this reason, more organizations are beginning to explore the use of machine learning as a means to more quickly and accurately identify threats.
Machine learning -- a discipline that emerged from research into pattern recognition and computational learning theory -- applies algorithms to data culled from systems and networks to make predictions about potential outcomes. In network security, it's used to profile traffic to recognize potentially dangerous threats.
Machine learning has been around for decades, but it has been prohibitively expensive because of its intensive computational requirements. However, the relative decline in processing costs and vast improvements in the algorithms used to spot trends are making it a much more viable option for businesses.
A number of security vendors -- including Cylance Inc., FireEye Inc. and Carbon Black Inc., as well as managed service providers such as Masergy Communications -- are leveraging advanced machine learning as a mechanism to accelerate threat identification for a number of use cases beyond network traffic profiling and anomaly detection. Advanced machine learning can be applied to analyze user behavior and detect insider threats. The technology can also be used for spam filtering, malware identification and detection.
With respect to network profiling, advanced machine learning can be used to recognize patterns in network flow, dig through historical data to identify trends and spot issues indicative of a potential threat. The most comprehensive tools ingest data from multiple sources, including network flow, log analysis, signature detection, vulnerability analysis and threat intelligence.
Conceptually, one of the major advantages to using advanced machine learning for security is its ability to process and analyze huge volumes of data collected over time -- much faster than humanly possible. In an era where almost all businesses suffer from a shortage of human security resources, this can be a tremendous help in ferreting out the issues that should command the highest-priority attention.
Challenges still exist
That said, machine learning needs some fine-tuning before it can accurately detect the most urgent network security problems. First off, establishing a baseline of what is normal on a network is next to impossible in environments where virtually every network is already compromised. Then, there is the ongoing challenge of constantly shifting user behavior and ongoing changes in system-produced traffic. These changes are likely to produce red flags where there really are no significant issues. So, again, IT must sort out more false positives and spend time away from shutting down the real threats.
Clearly, there is enough progress -- and promise -- in using advanced machine learning to find the proverbial needle in the network haystack. It is worth exploring as an option, provided organizations understand its current limitations. It is critical to know, like all things security-related, there is no one silver-bullet cure for what ails the enterprise. Instead, organizations need to use machine learning in conjunction with multiple tools and human resources. There is, after all, no substitute for experience.
Advanced machine learning business benefits
Machine learning security from a Dell perspective
Machine learning still faces some hurdles