Browse Definitions :

canary testing

What is canary testing?

In software testing, a canary is a push of programming code changes to a small group of end users who are unaware that they are receiving new code. Because the canary is only distributed to a small number of users, its impact is relatively small and changes can be reversed quickly should the new code prove to be buggy.

For rollout of incremental code changes, a canary release of new features or even a new version of the software enables the development team to quickly evaluate whether the new software release provides the desired outcome. Canary deployment tests, which can include automated testing, are included in the canary release after testing in a sandbox environment has been completed.

Canary deployment enables developers to migrate a small subset of users to new functionality in a new release. This minimizes the effect of potential issues related to new software and makes it easier for the development team to roll back a buggy release of new functionality when only a fraction of the overall user base is exposed to the new code.

The word canary describes the code push to a subset of users because canaries were once used in coal mining to alert miners when toxic gases reached dangerous levels. Like the canary in a coal mine, the group of users selected to receive new code in a canary test is unaware they are being used to provide an early warning.

Canary testing new code on a small subset of users
In canary testing, a small group of end users serve as a test group for code changes. If a change causes problems, monitoring software alerts the IT team so code can be fixed before it's released to a larger group of users.

Deploying software changes

The canary deployment strategy is just one of several techniques for updating software being used in a production environment. Ideally, new software deployment techniques should enable real users to have access to the most up-to-date version of the application while also minimizing any negative effects on user experience. Developers must balance the need for reducing downtime due to bugs in updated software while also reducing user base issues related to using out-of-date software.

Some other methods for deploying software updates include the following:

  • Basic deployment strategy. Also known as the recreate approach, this strategy updates all systems with the new software at the same time. This is the simplest deployment strategy, which means it can be the easiest for software development teams to use, but it can also be the riskiest. If the upgrade is flawed, the entire user base will be affected. Remediating a buggy release can be disruptive, especially if the bugs are not discovered until after all users have been updated. This means the buggy code must be removed and if a patched version is not immediately available, all users must roll back to the prior release.
  • Incremental or rolling update strategy. This is similar to the basic deployment strategy, except it partitions the user base into smaller groups that receive the update in phases. This approach enables the deployment team to halt a release before it affects the entire user base in the event there are issues with the new release.
  • Blue/green deployment strategy. This calls for deploying the new version of the software alongside the old version. Blue/green deployment requires double the resources of the incremental deployment strategy, but it reduces the effect of a buggy release because users can be switched back to the current release with minimal downtime.
  • A/B testing deployment. This is used to test specific features by rolling them out to specific users based on metrics. For example, A/B testing metrics may reflect specific types of use, localization, browser, screen size or operating system.

Enterprises and other large organizations can mix and match these and other approaches to optimize their continuous delivery and continuous integration strategies. This includes assessing customer needs for the ability to use the most reliable version of the software being deployed -- or the ability to use a newer version of the software that includes new features or functions that solve specific problems for the customer.

How does canary testing work?

Canary testing works in the following way:

  1. The development team selects which unwitting customers will be testers. This is usually a small subset of all customers but a large enough group to produce statistically meaningful results.
  2. The team members set up a testing environment that operates in parallel with the existing environment.
  3. They configure the system load balancer to route user requests from designated canary testers to the new environment.
  4. They initiate the canary test by routing tester requests to the new environment.
  5. They monitor testers for as long as it takes to ensure the new version is operating as expected. If the new version meets deployment criteria, the software can be released to all. If the new version fails due to bugs, poor performance or some other issue, canary testers are rerouted to the original version of the software.

Canary deployment is important, but it's not the only good choice for managing deployment strategies. Learn more about when to use a canary vs. blue/green or rolling deployment.

This was last updated in May 2022

Continue Reading About canary testing

  • virtual network functions (VNFs)

    Virtual network functions (VNFs) are virtualized tasks formerly carried out by proprietary, dedicated hardware.

  • network functions virtualization (NFV)

    Network functions virtualization (NFV) is a network architecture model designed to virtualize network services that have ...

  • overlay network

    An overlay network is a virtual or logical network that is created on top of an existing physical network.

  • encryption

    Encryption is the method by which information is converted into secret code that hides the information's true meaning.

  • X.509 certificate

    An X.509 certificate is a digital certificate that uses the widely accepted international X.509 public key infrastructure (PKI) ...

  • directory traversal

    Directory traversal is a type of HTTP exploit in which a hacker uses the software on a web server to access data in a directory ...

  • resource allocation

    Resource allocation is the process of assigning and managing assets in a manner that supports an organization's strategic ...

  • chief digital officer (CDO)

    A chief digital officer (CDO) is charged with helping an enterprise use digital information and advanced technologies to create ...

  • security audit

    A security audit is a systematic evaluation of the security of a company's information system by measuring how well it conforms ...

  • implementation

    Implementation is the execution or practice of a plan, a method or any design, idea, model, specification, standard or policy for...

  • first call resolution (FCR)

    First call resolution (FCR) is when customer service agents properly address a customer's needs the first time they call.

  • customer intelligence (CI)

    Customer intelligence (CI) is the process of collecting and analyzing detailed customer data from internal and external sources ...