Getty Images
Automated testing is not a magic bullet
Test automation rarely serves as a shortcut to faster testing. Its greatest value lies in improving software quality, expanding coverage and enabling testers to focus on high-value work.
Since the beginnings of automated testing in the 1970s and 1980s, the IT industry has been wrestling with the question of how test automation can be a magic bullet that accelerates the software development lifecycle.
A recent Omdia survey revealed that our industry continues to struggle with this. Responses to two questions specifically elucidate the issue:
- In response to the question: "Which of the following application development processes in your organization consume the most time?” QA and testing were the answer of 34% of respondents, and ranked second highest, below security.
- In response to the question: "Into which of the following application development processes has your organization integrated automation tools or techniques?" Testing and QA was the process most frequently cited.
These responses show that although organizations are focused more on automating testing than any other part of the software development process, testing remains one of the most time-consuming parts of the process.
There might be reasons for QA being the most time-consuming process in the development cycle that are unrelated to test automation, including:
- High developer to tester ratios.
- Collaboration challenges across distributed teams.
- Highly complex code.
- Highly complex domains.
Yet it seems that the investment in test automation has not produced the desired increase in velocity of testing. This suggests that test automation might not have proved to be a magic bullet.
This article will delve into the reasons why test automation might not significantly reduce test cycle time. We’ll look at the challenges involved in test automation, understand the types of automated testing and why not all tests can or should be automated. We will also look at why some approaches to test automation actually increase time spent on testing.
Finally, we’ll address the question of test automation as a magic bullet. Although the survey results show that automation has produced disappointing results in terms of increasing testing velocity, how does test automation affect the overall quality of the product? How does well-designed test automation affect the efficiency and effectiveness of the test team? Should our evaluation of the value of test automation go beyond its affect on test velocity?
Why test automation may not reduce test cycle time.
There are several reasons that test automation might not reduce overall testing time. These include what and how we choose to automate. Let’s look at the main reasons
What we choose to automate.
There are many types of test automation. We can automate almost all types of testing including API, smoke, sanity and regression as well as continuous testing across DevOps automated pipelines. Test types including non-functional, accessibility and mobile are candidates for automation. More recent advances include testing AI using automation. We are also now vibe testing; using AI to generate and maintain automated tests.
Sometimes we attempt to implement automation into everything that can be automated regardless of whether it should be. This often leads to ineffective automation chaos, increasing overall test time. In addition, there are some types of testing such as user experience, that must be manual.
How we chose to automate.
Often, inefficiencies in test automation begin with the automation approach. We focus test automation efforts on test types that are the most inefficient to automate. Although almost all types of testing can be automated, not all should be automated. Instead of creating the test automation pyramid, we create the antipattern, the ice cream cone.
To get the most value from test automation, it is important that the high levels of automation should be at the unit, component and API levels with less focus on the GUI tests, i.e. the test pyramid. This focus not only finds defects early in the process; but also, reduces the overall automation suite maintenance requirements. Unit tests are quick and easy to automate whereas GUI automation is more complex and requires continual maintenance.
When we create massive regression suites while limiting unit, component and integration testing, we create the ice cream cone. This massive maintenance effort required for regression suites can create automated testing processes that take longer than the manual tests themselves. Also, all automated testing must be balanced with human testing to ensure the results are accurate and edge cases are not missed.
Why isn’t test automation a magic bullet?
Even with a well-designed test automation strategy that includes focusing automation early in the development process, carefully considering test automation maintenance time and cost, and implementing automation test optimization, organizations are not always satisfied with the return on investment in terms of time savings.
Often this happens because organizations expect test automation to replace manual testing. To address this dissonance between cost and time savings, we should consider that test automation offers benefits beyond saving time.
If we view test automation as tool in the pursuit of quality rather than a replacement for manual testing, we can set more realistic expectations for its role in the test process. If we consider that test automation servers a broader purpose -- which is to improve the quality of the product -- the ways in which we evaluate success changes.
When we evaluate test automation based on its contribution to overall product quality, we consider the role of test automation increasing test coverage, in finding critical defects, and in increased defect removal efficiency.
These improvements in quality happen because test automation allows for much greater test coverage that can run more frequently. If the highest levels of automation are focused on unit and component testing, we find defects earlier when they are most easily remediated. Regression test suites for end-to-end testing greatly expand test coverage, finding defects that might be missed when we limit the number of tests in the interest of speed.
Most importantly, when we consider test automation as tool for improving quality, we look to our testers to optimize its use. Testers write the test scenarios that are coded and then executed through automation. In this way, automation frees up testers’ time, which gives them the opportunity to focus on creating innovative test designs and creative test cases. It gives them the time not only to fully vet automation results, but also, to explore new ways to improve quality by employing new tools and techniques that incorporate AI. Finally, test automation gives testers the time to focus their attention on the aspects of testing that only a human can do.
If we look at the value of test automation as a tool that enables testers to focus on their most important responsibilities and exercise their greatest strengths in innovative test practices, it might just be a magic bullet.
Gerie Owen is a QA engineering manager at Roobrik. She is a conference presenter and author on technology and testing topics, and a certified Scrum master.
