9 CI/CD best practices for DevOps teams
Building and maintaining a CI/CD pipeline requires more than just chains of automation. Adopt these CI/CD approaches to maximize your development and deployment efforts.
Through my experience in building and maintaining CI/CD pipelines, I've identified CI/CD best practices that may help your CI/CD pipeline create true value through improved software quality with less time spent on manual tasks.
Remember that each project is different and some tips may work better than others for your application. Always do what makes sense for your project.
1. Set priorities for security, testing and time to release
A CI/CD pipeline may provide more secure software, more automated testing or quicker time to release. If you have a slow, manual process in place, you might be able to achieve all three -- but more than likely, these improvements will conflict with each other. To get the most value out of your pipeline, prioritize the most important benefit and focus on tools and processes that grant that benefit.
For example, if your main goal is to release software more quickly, but you also want to improve security, consider creating separate automated processes in your pipeline for each. You can have a primary pipeline that builds, tests and releases your software as you develop new features. Alongside this pipeline, where you won't slow down the main development, you can also run thorough automated security scans of your applications to spot vulnerabilities.
2. Define specific goals for your CI/CD pipeline
Before you begin your project, ask this question: "What do I hope to accomplish by using a CI/CD pipeline?" The answer might be shorter time to release, better software quality or increased automated testing, but these goals should all be part of a checklist for your CI/CD adoption.
Clear goals also inform all members of the team and provide a reliable North Star to keep everyone focused on choosing tools and creating automation that accomplish these goals. The team also can better estimate the work needed to achieve those goals.
3. Use your pipeline as a way to foster collaboration
Despite the emphasis on automation in a CI/CD pipeline, it won't start out completely automated without some manual intervention. Changes may take weeks to months of work to add. Moreover, DevOps concepts such as continuous integration and continuous deployment are a mindset -- staff members need time to adjust to the new processes from a technical standpoint, but also to see the benefits and gain trust in automation.
Encourage the team to share ownership of the CI/CD pipeline. It's easy to fall into siloed patterns in which developers throw code over the fence to testers who then toss the code to an operations team for deployment. Without shared responsibility, bugs take longer to fix and overall quality decreases. When tests fail in your CI/CD pipeline, have developers triage the failures and correct the tests or the code, if needed. Testers should help guide developers on best practices, but it's more important to share the burden of maintaining the tests.
4. Choose tools that support your priorities
There are many tools available to help build your application -- automated UI testing, security testing and more. To start, automated build systems provide a remote, ephemeral environment where you can build and test your application code, or run any other automated tool that works in a command-line system. Popular examples include Travis CI, Jenkins, CircleCI, GitHub Actions and GitLab CI/CD. All of these use a configuration file, where you specify commands to be run for your application's pipeline. They also integrate with Docker containers, which ensures your build environment is properly set up with the dependencies needed to build and test your application.
Some testing tools, such as BrowserStack and Sauce Labs, provide remote machines specifically for running tests, which can speed up your test suites significantly. If compatibility testing is of high priority, tools such as these make it easy to run the same test across many different devices, screen sizes and platforms. Other widely available automated testing tools (Cypress is a personal favorite) also provide a testing framework, as well as a remote environment, to run the tests.
All these tools can be an economical way to fortify your pipeline without writing a lot of custom code. Remember that it's often best to use an already existing solution, rather than take the time and energy to create something from scratch.
5. Include the entire team in CI/CD implementation
Rather than have a small team implement your CI/CD pipeline, encourage all members who interact with the CI/CD pipeline to contribute something. Many small tasks arise throughout the process, including feature requests once a team gets familiar with the pipeline implementation. Group involvement helps introduce the team to the CI/CD process from the start and avoids a situation where development is slowed or stopped by something that only a handful of people can fix.
Developers, testers and DevOps engineers working together are best suited to find and implement automation to replace or augment processes and ensure the pipeline provides value. Collaboratively design a pipeline, from a high level, then dole out the work evenly among teams. DevOps engineers should provide guidance on best practices, but at the end of the day the CI/CD pipeline serves the development/QA team as its ultimate user -- and the more ownership the team has, the tighter the feedback loop will be between bugs and fixes.
6. Build once, test repeatedly
One way to create a more efficient CI/CD pipeline is to create a project's build artifacts only once. For example, if your application deployment uses a Docker container, build the Docker container at the beginning of your pipeline and reuse the same container for all subsequent deployments and testing. Not only will this save time, it also ensures continuity across deployments and avoids any issues with dependencies pulled in during the Docker build.
7. Mock services in development environments
One item that can hinder tests is rate limits imposed by third-party services. For example, an application uses a third-party authentication service such as Amazon Cognito or Auth0 to register and authenticate users. In production, these services work well because they are expected to handle production levels of load. In development, however, these services' endpoints are usually rate-limited to avoid costs associated with handling production levels of load. Consider Stripe's free "test" mode -- the company still provides bandwidth (and incurs costs) to respond to requests from this service, but without rate limits it would be uneconomical for the company.
You could pay for the same level of service in your development and production environments, but you'll pay at least twice as much to cover them both. A more economical strategy is to mock these third-party services and provide a feature flag to enable a fake implementation of the service. You won't face rate limiting, the service will always respond successfully and your load and performance tests can more accurately measure your code's actual performance. Don't forget to test with the real integration before shipping to production.
8. Use ephemeral, scripted environments
It is possible to have a CI/CD pipeline with static environments, where builds are installed as a part of CI/CD, but dynamic environments make more sense. Having one environment in CI/CD means you can deploy and test only one build at a time. Dynamic environments enable your pipeline to scale to as many builds as you would like. Tools such as Docker Compose or Kubernetes make dynamic environments faster to configure and launch compared to cloud compute servers alone.
9. Fail fast
Design your pipeline in such a way that you can quickly find major defects. Collect a number of tests that validate your application's most important features and run them on new deployments, prior to more intensive testing efforts. This way, you invest a minimal amount of time to confirm a deployment is worth testing more thoroughly. Defects found soon after development are often easier to remediate than those discovered after developers have moved on to other tasks.
Building a CI/CD pipeline is an ever-evolving process, with many different paths. It is certainly worth the effort to build a pipeline -- even more so, a pipeline that is efficient and valuable. Following these tips will help you create a process that improves the quality of your projects and saves you time.