This content is part of the Essential Guide: How to define SIEM strategy, management and success in the enterprise

Security information management systems aspire to real-time security

Today’s security information management systems (SIM) are excellent forensics tools, but they haven’t yet achieved status as effective real-time security tools.

Big data is coming to information security. And it’s forcing security managers to take a critical look at their existing technology investments, in particular data collection points such as security information management (SIM) systems, to determine if they’re up to the task of helping with real-time security analysis of event data.

In general, the industry has needed to manage expectations on real time. Deep-packet inspection technologies set the tone for prevention, and they expect the same level of situational awareness from SIM, but don’t tend to get it.

Joe Gottlieb, president and CEO, Sensage 

Enterprises need to understand what’s happening on networks in as close to real time as possible. Yet experts agree that real-time analysis might be a bit ambitious at this juncture, for SIM especially. Security teams should temper their expectations of what “real time” means, what SIM and other analytics technologies are capable of, and the resources needed to observe and react to security incidents in real time.

“It is ambitious, especially for SIM, because the event has to happen to get logged, sent to the SIM or log aggregator and run through the rules engine,” said Diana Kelley, founder of Security Curve, a consultancy in New Hampshire. “All of this takes time and it’s not real time. You’re not looking at live traffic like an IPS or next-generation firewall would. That’s closer to real time than an event going through a log management system that parses the data and sends it to a SIM where correlation rules are run.”

Actionable information has always been the pot of gold at the end of the SIM rainbow, but finding the treasure often gives way to painful rule writing and integration exercises. SIM rules can be a hardship because, like all signature-based defenses, security teams need to understand what they’re looking for in order to establish proper alerting thresholds.

“If your thresholds are too high, you’re not alerted quickly enough,” Kelley said. “If they’re too low, your SIM is slamming you with alerts. Figuring out those thresholds is what makes rule writing so complicated.

Eye On SIEM Systems:

Editor’s Note: This news story is part of's "Eye On" series that brings together various perspectives on security topics throughout the year from SearchSecurity and its sister sites. In the month of March the series examined SIEM systems.

Log management or SIM can be great for forensics—going through and finding needles in haystacks,” Kelley added. “If you don’t know where to start, it gets problematic.”

The end result has often been frustration with the product; companies sometimes end up shutting off the analytics and are left with a compliance and reporting tool that in some cases may have cost more than six figures to buy, install and maintain. However, using SIM to its fullest may not be a luxury for much longer. Not only do regulations require log analysis and reporting tools, but the crush of targeted, persistent attacks against high-value government, manufacturing and financial targets could inject renewed interest in maximizing SIM investments.

Extended security information management system capabilities for real-time security
Robert Capps, senior manager of trust and safety at online ticket broker StubHub, augmented his company’s SIM and monitoring technologies with fraud detection technology from Silver Tail Systems that looks for anomalies in how users interact with the site versus a baseline of normal traffic. He cited frustration with the inability of SIM and other network security devices to pick up abuses of legitimate StubHub services perpetrated by attackers. Intrusion prevention systems (IPS), for example, saw only legitimate network traffic, while SIM recorded successful logins with legitimate accounts created by attackers for the purposes of fraud.

Capps said he believed IPS, SIM and other analytical tools weren’t effective at analyzing security events, but didn’t have the data to support it. By taking a real-time analytics approach, he said he was able to identify problems and change his company’s security response without changing the customer experience.

“IPS is great if someone is trying to attack your firewall; it’s not real good at identifying bad actors who are getting in with good traffic, especially if they’re using your Web application like everyone else,” Capps said. “I’d rather have a tool that says, ‘This looks odd and doesn’t fit with my transaction flows.’ That was the direction I needed to identify zero-day attacks.”

Leading SIM vendors such as ArcSight, an HP company, Sensage and Q1 Labs (IBM) are talking about extending the capabilities of their products in the direction of business analytics and data warehousing in order to accommodate big data analysis, essentially bringing real time into the equation. Security analysts are burdened with a virtual landslide of data from not only network security devices, but operating systems, applications and even user behaviors. Sensage President and CEO Joe Gottlieb says his company’s tools already give organizations the option of pulling security data from particular sources into a data warehouse where correlation rules are run against smaller subsets of data flows.

“The data is no more than five minutes old,” Gottlieb said. “Real time is really about a mix of sources and freshness (of data). The least common denominator is the oldest data you have in a state machine. That indicates how real real-time is… In general, the industry has needed to manage expectations on real time. Deep-packet inspection technologies set the tone for prevention, and they expect the same level of situational awareness from SIM, but don’t tend to get it.”

Data overload threatens SIM’s real-time security ability
Clearly, as monitoring and reporting technologies move closer to real time and more data sources are involved, the complexity involved in querying and processing events and maintaining thresholds grows too.

“If it’s too ambitious, it could come back to what you are trying to accomplish. There’s no reason why any organization wouldn’t want to do real-time analysis, but you need to balance that with what your environment looks like and what real time means to you and how you want to manage risk,” said Michael Callahan, vice president of worldwide product and solution marketing for HP ESP.

ArcSight also is heading toward real time via improved analytics and correlation for its SIM. Callahan said customers want enhanced performance and scalability – faster analytics and correlation from more sources – as well as more context from security events and what’s happening in IT operations.

“The next piece is to broaden it to the entire organization; this gives you the opportunity to look at what is your business’ overall risk,” Callahan said.

Experts caution that enterprises need to narrow their real-time scope, understand their environments and what attacks mean to different parts of their IT infrastructure.

“Every security team is drowning in data;  another problem with real time is that it puts more data into a data overload situation,” said Mike Lloyd, chief technology officer at Red Seal Networks. “Enterprises are already drowning in way too much data, and building more sensors with more data is not a great path forward. Making the human scale along with the data so that we can take action is hard and another real-time problem.”

Red Seal’s products promise continuous visibility into an IT infrastructure by mapping interactions between security devices and highlighting access points that could be vulnerable. Lloyd said companies should avoid the temptation of over-investing in any area of security, such as analytics, at the expense of prevention or forensics.

“That’s a big mistake in the enterprise,” Lloyd said. “You can’t know everything at a high scale.”

Security Curve’s Kelley said SIM needs to provide better rule sets and intelligence on attacks to its customers.

“SIM is very strong in forensics and piecing events back together,” Kelley said. “And it’s good at alerting in near real time on simpler, less complex issues.”

Dig Deeper on SIEM, log management and big data security analytics