This content is part of the Essential Guide: How to define SIEM strategy, management and success in the enterprise

Some CISOs consider ripping out or augmenting outdated SIEM systems

Outdated SIEM systems were difficult to deploy and costly to maintain, according to one expert. Today, CISOs are considering highly integrated, lightweight systems with more automation.

Enterprises with older security information event management (SIEM) systems are taking a second look at their hardware, according to experts, and in some cases, businesses are mulling over whether to augment SIEM systems with additional tools, or rip-and-replace systems altogether.

They are right now trying to justify the continued license renewal to get more value out of it for the purposes of risk management and situational awareness.

Bill Sieglein, CEO, CISO Executive Network

Gregg Woodcock of communications services provider MetroPCS Wireless Inc., sees log correlation and analysis as an integral part of running an efficient and secure business.

In fact, the Dallas-based software engineer sees so much value in correlating and analyzing logs, he founded and chairs a Dallas-based user group devoted to Splunk, a search tool that can take in many types of log data, from customer transactions to network activity, and call-record data, correlate it, and analyze it to discover valuable intelligence. The tool has become so popular, according to Woodcock, that many of the members of the Splunk user group are at organizations that have security information and event management (SIEM) systems in place, but want to use Splunk as a Google-type search bar to augment them.

MetroPCS used Splunk to monitor for terms-of-service violators of its free international phone calling plan. Woodcock said users were instantly able to see where traffic was going and how much it was costing the company. People violating the terms of service for using the free international calling for business use, were detected quickly from the call log data and were cut off before expenses got out of control, Woodcock said.

“The amazing thing is the speed at which it can do the things it does and the insight it provides to everyone who uses it,” Woodcock said. “It is in essence, Google for your logs; it ingests them in real-time and time stamps them and then it allows you to do just about anything with it using a UNIX-like set of search commands.”

Eye On SIEM Systems:

Editor’s Note: This news story is part of's "Eye On" series that brings together various perspectives on security topics throughout the year from SearchSecurity and its sister sites. In the month of March the series examined SIEM systems.

Splunk added support for security monitoring in 2010. It can also generate alerts in real time. The fact that it is being used by hundreds of people to augment existing SIEM systems is a sign that many early SIEM deployments were either too complicated to configure correctly, or had too many constraints to get valuable intelligence from the system, Woodcock said. “With Splunk, you dump data in and impose ad hoc schemas on the data that may only be useful to you and go from sorting to search and it’s a radical change,” he said. “With many other products you have to do development to have a data schema that you can use.”

Currently, most SIEM systems are set up for their compliance and reporting capabilities, and many continue to be deployed to meet that minimum use case, said Bill Sieglein, CEO at the CISO Executive Network. Sieglein recently completed a series of roundtable sessions with Fortune 1000 CISOs on security operations, including log management and SIEM. He said many CISOs are wondering whether or not to rip and replace their outdated SIEM systems with newer SIEM technology to create an intelligence platform.

“In almost every case, the implementations were longer and more expensive than they originally anticipated,” Sieglein said. “They are right now trying to justify the continued license renewal to get more value out of it for the purposes of risk management and situational awareness.”

Early SIEM implementations were cumbersome to deploy, and took two to three years in some cases with three quarters of the cost going to professional services for deployment assistance, Sieglein said. Today, more lightweight systems are being considered – SIEM platforms from McAfee (NitroSecurity), IBM (Q1 Labs) and LogRhythm, which promise faster implementations and more out-of-the-box automation, Sieglein said.

For organizations that made a substantial investment in SIEM, many are sharing stories about how difficult the journey has been, Sieglein said. For businesses dedicated to reviewing logs, it took a large number of staff to not only watch for events, but also to manage the system so it isn't overwhelmed by the log data. The system had to be kept fully patched and someone needed to understand how to do specialized reporting in order to get value out of the system.

“There were complaints that SIEM 1.0 requires a lot of babysitting just from a systems perspective,” Sieglein said. “It didn’t allow for resources to be dedicated to staring at the glass and watching events. Now SIEM 2.0 promises faster implementation, a lot less system management where resources and time can be dedicated to using the analytics and actually taking action based on the types of alerts that they are seeing.”

Chris Petersen, co-founder and CTO of LogRhythm agrees that early implementations were sometimes nightmarish to deploy and maintain, and often were left running in a poorly configured state to meet a specific compliance mandate.

SIEM was initially designed to solve the massive amounts of data generated by intrusion defense systems by trimming the IDS data down to something that was more manageable and actionable, Petersen said. SIEM vendors made it more complicated by adding a fuller spectrum of log data from the network layer, the device layer and the application and database layers. The focus now is to better manage the data sources and automate the analysis. “The goal today is to detect a broader class of events from insider threats, sophisticated intrusions and deeply embedded breaches by making that forensic layer immediately accessible,” Petersen said.

SIEM vendors have learned that it’s not feasible to expect companies to do manual log analysis, he said.

“Nobody has the perfect solution; these are complex problems,” Petersen said. “What we do have today is more information to look at than we’ve ever had before. If we can analyze it correctly and creatively via different techniques… we put the intelligence into the system to point customers to places to go investigate and have a thorough experience to quickly arrive at a conclusion and course of action.”

To get something out of a tool you have to invest time, money and effort into people.

Bill Bradd, assistant division chief for the Office of Technical Security of Information Security, the U.S. Census Bureau

The late Eugene Schultz, a noted network security expert, warned in 2009 that SIEM vendors needed to address the complexity of installing SIEM. Schultz, a strong believer in the merits of SIEM technology, said “the availability of good technology is by no means any guarantee that people will buy it.” Most SIEM products require months of tuning after the initial installation, he wrote in a blog entry on why the SIEM market isn’t doing better. “One well-selling SIEM tool can require the installation and maintenance of four separate machines on the network and has so many functions that many levels of menu traversal are required to get to some of the most basic functions. Troubleshooting SIEM tools is generally no picnic, either,” he wrote.

Organizations considering a broad SIEM deployment need to have the ability to conduct a robust test and evaluation process of SIEM products, said Bill Bradd, assistant division chief for the Office of Technical Security of Information Security at the U.S. Census Bureau. It’s an investment in technology, but also people knowledgeable in maintaining and monitoring the system, Bradd said.

The U.S. Census Bureau has been building out the capabilities of its Sensage SIEM system from collecting about 150 systems about five years ago when the scope was primarily regulatory compliance to more than 2,800 network devices and servers today as part of a broader information security strategy. That meant acquiring new hardware to handle the massive amounts of log data, working with system owners to feed the data into the SIEM system, and a development team to create scripts to take in and parse the various system logs, Bradd said. The SIEM system can audit system log data from Unix and Linux servers, Windows event logs, network firewalls and routers and switches.

“The volume of data is always a concern,” Bradd said adding that tuning is always an issue, but that the Census Bureau was able to get it under control. “If you know you’ve got an application that is going to generate a certain kind of alert, it’s not a difficult process to tune that out.”

Bradd said the Census Bureau also plans to send alerts to system owners so a network engineer in charge of maintaining routers and switches can investigate an alert and report back within 72 hours whether it is an event that can be remediated internally or if it is a serious security problem that requires reporting and a full investigation. The built-out system has been used to find misconfiguration issues and detect malware infected machines and trace the malicious code back to the site the user visited, Bradd said. From server perspective, the Census Bureau is monitoring individual user activity to determine if an attacker is conducting a brute-force password attack on an employee account or if an employee has simply forgotten their password.

“To get something out of a tool you have to invest time, money and effort into people,” Bradd said.

There are signs that vendors have addressed some of the problems with earlier releases. The experience has been fairly smooth for one Canadian firm that deployed a newer LogRhythm system in February 2011. The firm, Cara Operations Ltd., which operates 650 restaurants, mostly corporate and franchise locations, deployed the SIEM system to monitor its payment systems for PCI DSS compliance. “We know it’s capable of doing more than PCI compliance, but ultimately PCI was behind the decision to move forward with it,” said Rik Steven, project manager information technology at Cara.

The company uses a managed security services provider (MSSP) to monitor the system and handle alerts. While the MSSP monitors the system 24 hours a day, an IT professional within Cara is acting as a threat analyst to monitor the system internally. Restaurants span across five time zones, so the use of the MSSP was much needed, Steven said.

A team rolled out the system in about two months, deploying software agents at the company’s various locations. At first there was too much information and Steven said the company had to do some tuning over several months to “dial-back” and focus only on the compliance. “It’s easy to get overwhelmed with the fact that it tells you so much information,” Steven said.

The plan is to expand the system overtime to generate more reports and use newer features that can proactively address problems it identifies, Steven said.

“It’s been a big investment to get this in so we want to make sure we get our money’s worth out of it,” Steven said. “There’s a great deal of information it can tell us and we’ve only scratched the surface on the reporting that can come out of it even with just the basic canned reports.”

Dig Deeper on SIEM, log management and big data security analytics