Building an infrastructure-as-code pipeline in the cloud
Understand the stages to manage infrastructure as code, from source control to activation deployment -- and how these functions can be accomplished through cloud services.
Developers have made containers, serverless functions and microservices the foundation of modern, cloud-native applications for many reasons -- but operational simplicity isn't one of them. The fragmentation of monolithic client-server or web-based systems into numerous microservices containers and cloud functions multiplies the number of components that IT staff must configure, deploy, update and monitor.
The underlying factors that compelled this shift to such a swarm of executables are standardization, integration and automation. Standardization enables developers to reuse the same code, image or template countless times with little effort. Integration enables processes to span software development to resource deployment. Automation allows these integrated, standardized components to be processed through software development and cloud operations pipelines, eliminating manual intervention and inadvertent mistakes.
Many labels and buzzwords describe such automated pipelines, but they can be boiled down to three categories:
- Infrastructure as code (IaC), which transforms cloud resource configurations into editable code that is compiled into deployed services.
- DevOps pipeline, or CI/CD pipeline, which defines the particular automation tools used to take application code from a source code repository to a deployable executable or cloud function.
- GitOps, which uses the Git source code management system to write applications, build pipelines, provision and manage resources (particularly Kubernetes clusters) and deploy applications.
What's in a DevOps or IaC pipeline?
End-to-end automation is an outgrowth of the DevOps mindset, processes and organizational structure. Regardless of whether one calls it IaC or GitOps, an operational CI/CD pipeline requires seamless integration of software development and IT operations teams.
In an event-driven pipeline, serverless functions can initiate many steps. These can be reduced to four primary categories or functions:
- A code repository for applications, microservices, serverless functions and IaC code;
- A build and test cycle to compile code into an executable, VM or container image, or IaC deployment and configuration steps;
- Release artifact staging and beta testing to stage and test compiled code, images and configurations without affecting production infrastructure in an environment where developers can rapidly iterate to fix bugs and roll back unsuccessful versions; and
- Release to production those artifacts that pass beta testing. The pipeline should support multiple release strategies such as blue/green, canary or rolling releases.
Although DevOps pipelines are commonly conflated with application development, those four steps also apply to IaC pipelines. Remember that code is the operative word in infrastructure as code. The goal is reusable, repeatable configurations, which requires a code repository to manage IaC descriptions, whether they are in YAML, Jinja or a domain-specific language.
This article is part of
What is configuration management? A comprehensive guide
As with application software, IaC means developers compile, integrate and test configurations on pre-production environments. Likewise, each step in an IaC process can be triggered by events, such as a code check-in to the repository or successful completion of syntax checks or beta test suites. The relevant tool then automatically performs the task.
How to create an IaC pipeline with cloud services
Cloud environments are well-suited for DevOps infrastructure-as-code pipelines for several reasons:
- The flexibility and mutability of virtual resources;
- The availability of managed, serverless pipeline services that don't require client-operated infrastructure or initial installation; and
- APIs, integral automation features and support for third-party automation tools that facilitate unattended triggers and execution of pipeline steps.
The following table compares DevOps IaC tools in each category across the major cloud services.
All of these cloud services provide documentation, interfaces and samples for integration with external IaC and CI/CD tools. Thus, an organization can build modern IT environments with IaC -- such as using Terraform to develop cloud-agnostic resource deployment scripts, and applying them to any of the cloud environments.
IaC pipelines: Adaptable to many situations
A chief benefit of DevOps pipelines is their flexibility to suit almost any situation. IaC pipelines are no different.
This example from the AWS DevOps blog illustrates the concepts and workflow, which includes the following steps:
- A developer changes IaC code and commits it to a repository, CodeCommit in this case, but often Git or Git service.
- A workflow manager like CodePipeline or event-driven function (Lambda, Azure Functions) detects the commission and triggers a code build.
- The build engine, whether CodeBuild, Terraform or another IaC tool, instantiates a compute environment and runs the specified build commands.
- The IaC deployment engine -- in this example it's CloudFormation, but it could be Terraform, Ansible, Deployment Manager or a similar product -- provisions the build environment resources and runs a validation check on the configuration.
- If the infrastructure configuration is validated, the IaC processes the configuration file and builds the environment.
As this example illustrates, all the tools required in an infrastructure-as-code pipeline are available as serverless cloud services that don't require dedicated infrastructure. They can be quickly set up and scaled to meet any requirement. A further advantage of cloud-based tools is their tight integration with the infrastructure, database and application services, eliminating the complexity of networking and interfacing with multiple systems.
