kentoh - Fotolia
How to approach IT logging in the cloud vs. on premises
IT organizations must monitor cloud infrastructure and apps with a process that's different than they're used to for on-premises environments. Learn how to do cloud logging here.
Few things are more important for IT infrastructure than proper logging practices. Logs from both application and infrastructure components help IT admins find the root causes of issues. With the help of machine learning to analyze logs, IT admins can prevent and remediate security issues.
Logs are everywhere and new ones are generated all the time, but they cannot be taken for granted. As enterprises move into the cloud, they must understand how cloud operations change logging procedures. This article covers many aspects of logging in the cloud, including the cost of log collection, how to approach different types of cloud logs and storage decisions.
Log collection in the cloud
Logging on site is normally straightforward in terms of collection and analysis. While the cloud doesn't change that in theory, it does change a few considerations.
Log storage and analysis with on-premises resources traditionally isn't expensive. Large drives with multicore CPUs are reasonable in cost, and often last for years. With log storage and analysis in the cloud, users pay for resources they consume. While logs are not typically massive files, the volume of logs that accumulates when you collect them from many sources can add up rapidly. Consider whether you'll be throwing money away on the monthly charge to store them.
The real surprise admins get with logging in the cloud is how log files are processed. Compute resource use becomes another ongoing monthly cost -- unless an IT organization disposes of unnecessary files, their costs will add up quickly. A third-party service for log analysis can help with this particular cost -- and it can work with both cloud and on-premises logs. However, log analytics tools add their own cost and time investment, which organizations must keep in mind.
Make savvy logging decisions
IT organizations must decide what to do with logs. Unlike with on-premises deployments, the IT organization does not control all of the infrastructure and components to run the application. The question is whether the organization has the ability to make changes in the cloud service. The answer depends on the service -- infrastructure (IaaS), platform (PaaS) or software (SaaS) -- as the access levels vary greatly.
With PaaS and IaaS cloud setups, IT admins have some control over logs and log analysis. With SaaS, the cloud provider controls almost everything from the data center hardware to the application. More admin control over the cloud environment makes it easier to see the value in log investment, because an organization can make meaningful changes, rather than just observing. The downside of more control over the cloud, with PaaS or IaaS, is the increased costs that go with it, which add to the cloud logging bill.
Pulling and processing detailed logs on a SaaS application might not be worth the investment in time and money, as an IT organization probably cannot make any effective changes to the SaaS environment or application. There's no real reason to work on these insights into information the cloud provider should already have.
With IaaS, cloud adopters have the most control over the environment, but are still limited to what the cloud provider allows them to adjust. Information from logs is key for cost-affecting decisions, such as whether to increase network bandwidth or compute resources for an application or service. Log analytics can lead to environment optimizations that pay off with lower overall cloud consumption.
When managing a cloud environment, select which logs to keep and which logs to ignore -- or not track at all. Look at the logs for the services that affect the environment and that are for aspects of the setup that you can change. Learn what the cloud provider monitors and optimizes as part of its responsibility for the environment.
To get started with logging in the cloud, look at the top 10 log types your organization monitors in its on-premises environment. Examples include access made between applications stacks, security logs or logs that show application errors. These logs should be universal -- from on premises to cloud services. Two additional logs that might not be on the on-premises log list are internet access and WAN. Connection to the cloud is key, so these logs should be a cornerstone of the collection process.
For logs the cloud provider would monitor, such as those tracking hardware events, verify the responsibility with the cloud provider. Commonly, the IaaS provider manages the hardware and therefore collects, analyzes and acts on hardware-related logs.
Plan to store logs
The length of time your organization retains its logs matters, especially for security events. After a security breach or attack occurs, logs are critical evidence used for a full investigation and to understand the effect of the issue.
Since cloud storage is a monthly cost, determine what logs to keep long term and which ones to dismiss on a more frequent basis. Consider transferring some types of cloud logs into long-term archival storage for future reference. Retrieval time can take hours or even days -- but this history could save a lot in costs when you need it.
While cloud-based log collection, management and analysis follow many of the same guidelines as these activities in on-premises environments, your organization will be most efficient if it accounts for cloud-related differences. Pay attention to log retention times, the size of your log storage and how many logs you're processing to keep costs under control. With logging in the cloud, key events and actions should be similar to those used in on-premises procedures, but don't forget to include WAN and internet activity logs, as these are more unique to cloud environments.
