What is canary testing?
In software testing, a canary is a push of programming code changes to a small group of end users who are unaware that they are receiving new code. Because the canary is only distributed to a small number of users, its impact is relatively small and changes can be reversed quickly should the new code prove to be buggy.
For rollout of incremental code changes, a canary release of new features or even a new version of the software enables the development team to quickly evaluate whether the new software release provides the desired outcome. Canary deployment tests, which can include automated testing, are included in the canary release after testing in a sandbox environment has been completed.
Canary deployment enables developers to migrate a small subset of users to new functionality in a new release. This minimizes the effect of potential issues related to new software and makes it easier for the development team to roll back a buggy release of new functionality when only a fraction of the overall user base is exposed to the new code.
The word canary describes the code push to a subset of users because canaries were once used in coal mining to alert miners when toxic gases reached dangerous levels. Like the canary in a coal mine, the group of users selected to receive new code in a canary test is unaware they are being used to provide an early warning.
Deploying software changes
The canary deployment strategy is just one of several techniques for updating software being used in a production environment. Ideally, new software deployment techniques should enable real users to have access to the most up-to-date version of the application while also minimizing any negative effects on user experience. Developers must balance the need for reducing downtime due to bugs in updated software while also reducing user base issues related to using out-of-date software.
Some other methods for deploying software updates include the following:
- Basic deployment strategy. Also known as the recreate approach, this strategy updates all systems with the new software at the same time. This is the simplest deployment strategy, which means it can be the easiest for software development teams to use, but it can also be the riskiest. If the upgrade is flawed, the entire user base will be affected. Remediating a buggy release can be disruptive, especially if the bugs are not discovered until after all users have been updated. This means the buggy code must be removed and if a patched version is not immediately available, all users must roll back to the prior release.
- Incremental or rolling update strategy. This is similar to the basic deployment strategy, except it partitions the user base into smaller groups that receive the update in phases. This approach enables the deployment team to halt a release before it affects the entire user base in the event there are issues with the new release.
- Blue/green deployment strategy. This calls for deploying the new version of the software alongside the old version. Blue/green deployment requires double the resources of the incremental deployment strategy, but it reduces the effect of a buggy release because users can be switched back to the current release with minimal downtime.
- A/B testing deployment. This is used to test specific features by rolling them out to specific users based on metrics. For example, A/B testing metrics may reflect specific types of use, localization, browser, screen size or operating system.
Enterprises and other large organizations can mix and match these and other approaches to optimize their continuous delivery and continuous integration strategies. This includes assessing customer needs for the ability to use the most reliable version of the software being deployed -- or the ability to use a newer version of the software that includes new features or functions that solve specific problems for the customer.
How does canary testing work?
Canary testing works in the following way:
- The development team selects which unwitting customers will be testers. This is usually a small subset of all customers but a large enough group to produce statistically meaningful results.
- The team members set up a testing environment that operates in parallel with the existing environment.
- They configure the system load balancer to route user requests from designated canary testers to the new environment.
- They initiate the canary test by routing tester requests to the new environment.
- They monitor testers for as long as it takes to ensure the new version is operating as expected. If the new version meets deployment criteria, the software can be released to all. If the new version fails due to bugs, poor performance or some other issue, canary testers are rerouted to the original version of the software.
Canary deployment is important, but it's not the only good choice for managing deployment strategies. Learn more about when to use a canary vs. blue/green or rolling deployment.