Browse Definitions :
Definition

flaky test

A flaky test is an analysis of web application code that fails to produce the same result each time the same analysis is run. Whenever new code is written to develop or update computer software, a web page or an app, it needs to be tested throughout the development process to make sure the application does what it’s supposed to do when it’s released for use. Logically, when put through the same test over and over, the code will produce the same result -- the application will either work properly every time, thus passing the test, or fail to work properly every time, thus failing the test.

However, seemingly at random, occasionally the same test of the same code will produce different results. Sometimes it will show that the code passed the test and the application worked as planned, and sometimes it will show that the code failed the test and didn’t work as planned. When the test fails to produce a consistent result, the test is deemed flaky.

Flaky tests can be caused by various factors:

  • an issue with the newly-written code
  • an issue with the test itself
  • some external factor compromising the test results

Once a test is deemed flaky, there are different approaches to dealing with the muddled results. Some developers will ignore the flakiness entirely, assuming that the issue is with the test and not with the newly-written code. Others will rerun their test multiple times and only go back to investigate further if the test fails a certain number of times in a row, indicating to them a true failure.

However, the safest approach -- the only way to truly find out whether there is a bug in the code -- is to halt the development of the application, fully investigate the cause of the flaky test and resolve it. If left unresolved and there truly is an issue with the code, one problem has the potential to wind up leading to another and another as more is built onto the faulty code.

When investigating the cause of a flaky test, the developer will need to gather data to try to discover differences within the seemingly random results in order to isolate the cause of the failed tests. The code should be re-examined, as should the test itself, and if no issues are found then external factors will need to be looked at to see if they might be at the core of the problem. The developer might look at whether the tests that passed were run at a certain time of day whereas the ones that failed were run at a different time of day, whether certain programs were running on the developer’s computer at the same time of failed tests that weren’t running when the tests passed or whether the tests that failed did so at the same point in the test or at different times during the test.

Sometimes, the cause of the flaky test is simple to diagnose and can be quickly fixed. That’s the best-case scenario. Other times, there is no easy fix, and though potentially costly and time-consuming, the developer may need to delete the test and rewrite it from scratch in order to ensure the accuracy of the test results.

Unfortunately, flaky tests are not uncommon -- Google, for example, reports that 16 percent of its tests show some level of flakiness. They can bring production to a temporary standstill, but they can be dealt with, and they can be resolved.

This was last updated in May 2019

Continue Reading About flaky test

Networking
  • local area network (LAN)

    A local area network (LAN) is a group of computers and peripheral devices that are connected together within a distinct ...

  • TCP/IP

    TCP/IP stands for Transmission Control Protocol/Internet Protocol and is a suite of communication protocols used to interconnect ...

  • firewall as a service (FWaaS)

    Firewall as a service (FWaaS), also known as a cloud firewall, is a service that provides cloud-based network traffic analysis ...

Security
  • identity management (ID management)

    Identity management (ID management) is the organizational process for ensuring individuals have the appropriate access to ...

  • single sign-on (SSO)

    Single sign-on (SSO) is a session and user authentication service that permits a user to use one set of login credentials -- for ...

  • fraud detection

    Fraud detection is a set of activities undertaken to prevent money or property from being obtained through false pretenses.

CIO
  • core competencies

    For any organization, its core competencies refer to the capabilities, knowledge, skills and resources that constitute its '...

  • change management

    Change management is a systematic approach to dealing with the transition or transformation of an organization's goals, processes...

  • IT project management

    IT project management is the process of planning, organizing and delineating responsibility for the completion of an ...

HRSoftware
  • recruitment management system (RMS)

    A recruitment management system (RMS) is a set of tools designed to manage the employee recruiting and hiring process. It might ...

  • core HR (core human resources)

    Core HR (core human resources) is an umbrella term that refers to the basic tasks and functions of an HR department as it manages...

  • HR service delivery

    HR service delivery is a term used to explain how an organization's human resources department offers services to and interacts ...

Customer Experience
  • martech (marketing technology)

    Martech (marketing technology) refers to the integration of software tools, platforms, and applications designed to streamline ...

  • transactional marketing

    Transactional marketing is a business strategy that focuses on single, point-of-sale transactions.

  • customer profiling

    Customer profiling is the detailed and systematic process of constructing a clear portrait of a company's ideal customer by ...

Close