Getty Images/iStockphoto
Why and how to export data from CloudWatch Logs to S3
There are several reasons to export data out of CloudWatch Logs and into S3. Use this step-by-step tutorial to automate the process.
Almost every IT admin uses log outputs to troubleshoot issues. Without proper logs, it's difficult to grasp why an application fails or why it's throwing warnings.
To gain value from logs, you need a central storage location that's cost-effective and helps you easily sort through the data. While Amazon CloudWatch Logs lets admins centrally store and monitor log data, exporting that data to Amazon S3 can be a beneficial option.
Why export data from CloudWatch Logs to S3?
Cloud admins typically use the CloudWatch service to ingest monitoring data from logs, applications and resources that run on AWS. CloudWatch Logs is a specific feature within CloudWatch that handles log data.
However, there are a few reasons to opt for a logging service other than CloudWatch Logs. For example, if you have a hybrid environment, you might want to use a monitoring or logging system that works across a broader range of resources. Alternatively, you might simply be more familiar with another logging tool or prefer another tool's log search and aggregation capabilities.
It can also get expensive to store logs inside of CloudWatch Logs. Since log data sits idle until admins need it, it makes sense to export that data to S3, which is generally a cheaper storage option.
Evaluate S3 storage classes
Depending on when and how often you access log data, you may want to consider a cold storage option. Although the S3 Standard storage class is cheap, it's hot storage, so it's more expensive than some alternatives. If you don't need fast and frequent access to the data, consider S3 Glacier, a cold storage service built for data archiving. It has a 99.999999999% retention rate, which means there's minimal risk of data loss.
How to export logs to S3
You can export log data to S3 using AWS Management Console, AWS Command Line Interface (CLI) or an SDK, as well as other methods via an API. You can export logs either automatically as they get ingested into CloudWatch or manually, depending on your needs.
Automation is ideal, as it eliminates the need to manually sign into the AWS console and export the logs from CloudWatch to S3.
Follow this step-by-step tutorial to export logs to S3 automatically, using AWS CLI.
Step 1. Create an S3 bucket using the code below. Title your bucket techtarget-bucket-92. This is the S3 bucket in which you'll store CloudWatch log data.
aws s3api create-bucket --bucket techtarget-bucket-92 --create-bucket-configuration LocationConstraint=us-east-2
Step 2. Set up access policies and permissions for the S3 bucket; by default, all buckets are private. The policy below gives CloudWatch access to export logs to S3.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:GetBucketAcl",
"Effect": "Allow",
"Resource": "arn:aws:s3:::techtarget-bucket-92",
"Principal": { "Service": "logs.us-east-2.amazonaws.com" }
},
{
"Action": "s3:PutObject" ,
"Effect": "Allow",
"Resource": " arn:aws:s3:::techtarget-bucket-92/*",
"Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } },
"Principal": { "Service": "logs.us-east-2.amazonaws.com" }
}
]
}
Step 3. Once the policy is created, set the policy on the S3 bucket:
aws s3api put-bucket-policy --bucket techtarget-bucket-92 --policy file://policy.json
Step 4. After the policy is set, start to export the logs from CloudWatch to S3:
aws logs create-export-task --profile ExportIAMUser --task-name "cloudwatchtos32022" --log-group-name "cloudwatchtos3" --from 1441490400000 --to 1441494000000 -destination "techtarget-bucket-92" --destination-prefix "log-output"
When this step is complete, you have successfully exported log data from CloudWatch to S3.
