Log monitoring refinements control data growth, costs

Log monitoring is more critical and cumbersome as cloud-native applications go mainstream, but subtle changes to log management and distributed infrastructure tools can pay big dividends.

For many enterprises, log monitoring has gone from a borderline luxury to a must-have way to troubleshoot complex microservices applications and software-defined IT infrastructures. That's because developers must comb through system logs to troubleshoot subtle problems in application code, and as IT resources become more ephemeral, highly granular log data captures information about individual systems that coarser methods don't catch.

However, as applications move from monoliths that run on a single server or high-availability pair to microservices that rely on multiple layers of virtual computing, the volume of log data generated by IT systems can quickly become unmanageable. Trying to keep historical log data on hand for incident reviews, forensic investigations or root cause analysis compounds the problem further.

"You never log less," said Bob Zoller, founding engineer at Good Eggs, a grocery delivery service in San Francisco. "We approached a point ... where we noticed we were spending an awful lot on [log management] and had to decide how much value we were getting out of it and if the numbers added up."

Bob Zoller

The firm, which has about 30 engineering employees, saw its log volumes increase from 10 GB per day in 2015 to more than 200 GB per day by 2018, which resulted in a bill for log management and analytics provider Sumo Logic of about $160,000 for the year.

"At the time, that was second only to our AWS bill," Zoller said.

The company took steps in 2018 to reduce the amount of data it stored with Sumo Logic for debugging. Instead, it streamed logs to a self-managed set of AWS S3 buckets, then transferred logs into Sumo Logic only when they were needed for debugging.

"If engineers wanted to do a query in Sumo Logic, they had to tell a chatbot what application and timeframe they cared about," Zoller said. "There were some things we had to give up, such as scheduled searches that were taking advantage of the fact that we were streaming all our logs into Sumo, but it was an OK tradeoff for us."

There were some things we had to give up, such as scheduled searches that were taking advantage of the fact that we were streaming all our logs into Sumo. ... You don't realize what you have until it's gone.

In late 2019, however, Good Eggs participated in the beta release of a new Sumo pricing tier for infrequently accessed data, which costs about a tenth of the $2 to $3 per gigabyte price for the full-fledged Sumo Logic service.

"You don't realize what you have until it's gone," Zoller said. "When we were streaming all of our logs in, you take for granted that everything is at your fingertips on demand any time you want to ask a question -- thinking about the application and timeframe you want before doing a search slows down developer productivity."

Infrequent access removes the additional step for developers to refine their search and load the right data into Sumo Logic from S3. It's still in beta, so Zoller can't yet compare the cost of the feature to the existing S3 system, but he hopes it will allow his company to resume sending all its log data to Sumo Logic at a cost more comparable to its self-managed workaround.

As the new log monitoring system moves to general availability, Zoller said he hopes Sumo Logic develops tools that let engineers assess whether a query is worth the cost to run ahead of time. Sumo Logic reps said this feature will be included when the product becomes generally available later this year.