prasit2512 - stock.adobe.com
Streamline multi-cloud cost analysis with AWS Data Exports and FOCUS
Uncontrolled costs drain budgets fast. AWS Data Exports and FOCUS reveal where your money goes, enabling precise analysis and optimizations for sustainable, cost-efficient growth.
Organizations need to execute cost monitoring and optimizations tasks regularly to identify and fix any unexpected high cost patterns as early as possible, especially in multi-cloud environments. Cost analysis tools are key to achieving optimal and sustainable cloud deployments.
Some of the most relevant optimizations my clients require are related to cost. Given the highly scalable nature of cloud infrastructure, high costs can easily occur in all types of environments, from early development tasks to production deployments, resulting in thousands of dollars of unnecessary spending.
AWS offers multiple tools to calculate, manage and optimize cost. In this article, I will focus on how AWS Data Exports, together with the support of the Financial Operations and Cost Usage System (FOCUS) data format, can help better manage costs across multiple cloud environments.
What is AWS Data Exports?
AWS Data Exports is a service that exports detailed billing and cost data from an AWS account to an Amazon S3 bucket in a standardized format. It enables organizations to execute a comprehensive analysis on which deployed components and usage patterns contribute the most to AWS cost and helps them find optimizations to reduce it.
It generates reports that are stored in S3 using either compressed .csv or Parquet data format. This allows data to be queried with SQL statements using data analytics services, such as Amazon Athena, or data can be visualized in tools like Amazon QuickSight.
What is FOCUS?
FOCUS was initially released in June 2024 by the FinOps Foundation, a non-profit organization focused on best practices related to financial operations in the cloud. FOCUS delivers a unified cost and usage data format for multiple cloud providers, including AWS, Azure, Oracle, Google Cloud and Databricks.
Given the rise of multi-cloud deployments across organizations, having a unified data format for FinOps analysis tasks is a very valuable feature that simplifies cost analysis and optimizations. With FOCUS, teams can analyze cost data from multiple cloud providers from a consolidated platform, while using the same processes and tools.
Export, query and visualize AWS cost data
AWS Data Exports enables the streamlining and automation of internal cost analysis processes by implementing custom tasks for very specific scenarios, which can be done using AWS SDK and pre-defined SQL statements.
When configuring AWS Data Exports -- which can be done from the AWS console, SDK or CloudFormation -- account administrators can choose up to 60 columns to be exported in daily reports. These columns comply with the FOCUS data format and are related to specific resources, services, usage types, usage units, cost amount, discounts, assigned invoices and tags.
Once this data is available in S3, it is possible to execute very detailed and specific SQL-based data analysis on how AWS cost is incurred and find optimizations and areas of improvement. This data can also be visualized using Amazon QuickSight, which simplifies the analysis of cost and usage information using a Business Intelligence interface that doesn’t require the execution of complex SQL statements. The combination of SQL statements and QuickSight allows a wide range of team members to have visibility into cost patterns and implement FinOps tasks and optimizations in a very efficient way.
AWS Data Exports use cases
The most common use case for Data Exports is detailed cost analysis. In many cases it is necessary to do a more detailed analysis of usage data, in order to find the root cause of high cost situations. For example, enterprises need to have the ability to identify the top resources by cost, do a data break-down at the resource level – such as an EC2 instance, ECS service, S3 bucket or RDS database -- and see the specific utilization patterns that generate high cost, such as compute resource allocation, data storage, license fees, requests and data transfer fees. This type of detailed analysis helps find and fix high cost patterns and configurations.
There are other tools in AWS that help with anomaly detection, optimization recommendations, billing alerts and notifications, as well as high-level visibility of cost and usage patterns. These tools include:
- AWS Billing Conductor.
- AWS Budgets.
- AWS Compute Optimizer.
- AWS Cost Explorer.
- AWS Cost Anomaly Detection.
- AWS Data Exports.
- AWS Pricing Calculator.
- AWS Trusted Advisor.
It is highly recommended to use Data Exports together with all available cost management tools in AWS as part of a solid and comprehensive cost optimization strategy that ensures cloud components contribute to sustainable business growth.
Ernesto Marquez is owner and project director at Concurrency Labs, where he helps startups launch and grow their applications on AWS. He enjoys building serverless architectures, building data analytics solutions, implementing automation and helping customers cut their AWS costs.
