Browse Definitions :
Definition

flaky test

A flaky test is an analysis of web application code that fails to produce the same result each time the same analysis is run. Whenever new code is written to develop or update computer software, a web page or an app, it needs to be tested throughout the development process to make sure the application does what it’s supposed to do when it’s released for use. Logically, when put through the same test over and over, the code will produce the same result -- the application will either work properly every time, thus passing the test, or fail to work properly every time, thus failing the test.

However, seemingly at random, occasionally the same test of the same code will produce different results. Sometimes it will show that the code passed the test and the application worked as planned, and sometimes it will show that the code failed the test and didn’t work as planned. When the test fails to produce a consistent result, the test is deemed flaky.

Flaky tests can be caused by various factors:

  • an issue with the newly-written code
  • an issue with the test itself
  • some external factor compromising the test results

Once a test is deemed flaky, there are different approaches to dealing with the muddled results. Some developers will ignore the flakiness entirely, assuming that the issue is with the test and not with the newly-written code. Others will rerun their test multiple times and only go back to investigate further if the test fails a certain number of times in a row, indicating to them a true failure.

However, the safest approach -- the only way to truly find out whether there is a bug in the code -- is to halt the development of the application, fully investigate the cause of the flaky test and resolve it. If left unresolved and there truly is an issue with the code, one problem has the potential to wind up leading to another and another as more is built onto the faulty code.

When investigating the cause of a flaky test, the developer will need to gather data to try to discover differences within the seemingly random results in order to isolate the cause of the failed tests. The code should be re-examined, as should the test itself, and if no issues are found then external factors will need to be looked at to see if they might be at the core of the problem. The developer might look at whether the tests that passed were run at a certain time of day whereas the ones that failed were run at a different time of day, whether certain programs were running on the developer’s computer at the same time of failed tests that weren’t running when the tests passed or whether the tests that failed did so at the same point in the test or at different times during the test.

Sometimes, the cause of the flaky test is simple to diagnose and can be quickly fixed. That’s the best-case scenario. Other times, there is no easy fix, and though potentially costly and time-consuming, the developer may need to delete the test and rewrite it from scratch in order to ensure the accuracy of the test results.

Unfortunately, flaky tests are not uncommon -- Google, for example, reports that 16 percent of its tests show some level of flakiness. They can bring production to a temporary standstill, but they can be dealt with, and they can be resolved.

This was last updated in May 2019

Continue Reading About flaky test

Networking
  • network management system

    A network management system, or NMS, is an application or set of applications that lets network engineers manage a network's ...

  • host (in computing)

    A host is a computer or other device that communicates with other hosts on a network.

  • Network as a Service (NaaS)

    Network as a service, or NaaS, is a business model for delivering enterprise WAN services virtually on a subscription basis.

Security
  • WebAuthn API

    The Web Authentication API (WebAuthn API) is a credential management application program interface (API) that lets web ...

  • Common Vulnerability Scoring System (CVSS)

    The Common Vulnerability Scoring System (CVSS) is a public framework for rating the severity of security vulnerabilities in ...

  • Dridex malware

    Dridex is a form of malware that targets victims' banking information, with the main goal of stealing online account credentials ...

CIO
  • audit program (audit plan)

    An audit program, also called an audit plan, is an action plan that documents what procedures an auditor will follow to validate ...

  • blockchain decentralization

    Decentralization is the distribution of functions, control and information instead of being centralized in a single entity.

  • outsourcing

    Outsourcing is a business practice in which a company hires a third party to perform tasks, handle operations or provide services...

HRSoftware
  • team collaboration

    Team collaboration is a communication and project management approach that emphasizes teamwork, innovative thinking and equal ...

  • employee self-service (ESS)

    Employee self-service (ESS) is a widely used human resources technology that enables employees to perform many job-related ...

  • learning experience platform (LXP)

    A learning experience platform (LXP) is an AI-driven peer learning experience platform delivered using software as a service (...

Customer Experience
  • market segmentation

    Market segmentation is a marketing strategy that uses well-defined criteria to divide a brand's total addressable market share ...

  • sales pipeline

    A sales pipeline is a visual representation of sales prospects and where they are in the purchasing process.

  • market basket analysis

    Market basket analysis is a data mining technique used by retailers to increase sales by better understanding customer purchasing...

Close