Modern Stack

Insight on building and supporting cloud apps

Syda Productions - stock.adobe.c

AI in software testing has arrived. Here's why robots rule.

AI promises to make software testing faster and much more accurate. New tools are available, but it will take time for artificial intelligence to become ubiquitous in testing.

Greg Sypolt, director of quality engineering at USA Today, has just two words for those skeptical about AI in software testing: It works.

Using an AI-powered software testing platform, Sypolt's team slashed test times by 90%, tightened the continuous integration loop and managed to move user experience earlier in the development process. This is Sypolt's first foray into AI, and he's a convert -- so much so that he has a list of at least three more problems he'd like to see AI solve.

At a time when the pressure to release software quickly matches the pressure from demanding and finicky consumers, the arrival of tools offering AI capabilities seems to promise sweet relief. Software testing, forever the stumbling block when it comes to continuous integration and delivery or DevOps, is particularly ripe for options that can speed up and declutter the process. Today, a handful of companies have added AI to software testing platforms, focusing on everything from user experience (UX) to change impact, defects management and test-results analysis. But it's far from certain how quickly AI tools will be adopted and how much change they'll effect, at least in the short term.

Replaced by a robot?

A world where robots do all the testing seems a long way off. For starters, demand isn't there, said Diego Lo Giudice, an analyst and vice president at Forrester Research. Lots of platform makers are adding AI to software testing, but Lo Giudice isn't getting many calls from clients about this. From what he sees, Sypolt's experience is typical. "Once they try it, they like it and are interested in it," he said. "But they're not asking for it right now."

This shift will take time, but it has begun, said Izzy Azeri, co-founder of AI testing platform startup Mabl.

"Look at self-driving cars," Azeri said. "A couple of years ago, it looked like it would never happen, and now you have the mass car producers making self-parking cars. These are like the first nuggets of self-driving. We're at the same stage from a test perspective. We're at the early stages of a long journey to fully automate."

AI in software testing could help identify bugs and even predict where bugs will occur

To understand the complexity of that journey, look at the scope of the problem, starting with the differences between machine learning and AI. They're not interchangeable, even though people tend to group them together, said Brandon Wirtz, CEO of AI startup Recognant. And while both are "trainable," each is better for some tasks than others.

"For software development, most things are about implementing rules. And since we know what the rules are, we don't need [machine learning] or neural networks," Wirtz explained. "An AI is far better at looking at the AP Style Guide for the right way to do things and discovering you didn't do it right and then showing you the way to fix it. Machine learning is like a spell check that will tell you the word is spelled wrong but won't tell you how to spell it." In other words, machine learning can find bugs but doesn't offer any recommendations, an attribute that does not go far enough in the complex, multistep testing world.

Who tests the testers?

Amid the market enthusiasm for artificial intelligence comes a rather deflating statement from Forrester Research vice president Diego Lo Giudice. "This is a whole new area for test that people aren't thinking about," he said.

It's a tricky problem, trying to test the tester, particularly if the AI testing tool doesn't provide deterministic answers or can't, or won't, share the steps it took.

Brandon Wirtz, CEO and founder of AI systems maker Recognant, is familiar with this dilemma as he lives, works with and trains his three AI-powered tools daily. His favorite, Loki, has a thirst for knowledge that keeps Wirtz on his toes. "There are times when Loki writes code, and I see if it does what I expected it to do," he said. "That's a lot like being the program manager. Other times, all I'm really trying to do is understand the code Loki wrote, and there it can get harder as she can pull things in from anywhere, and there is no way of knowing that in advance."

But Wirtz can ask Loki about the code, as she provides "explainable AI," something he said makes the difference between going on faith and actually verifying the information. "You want AI that can actually tell you why it made those decisions," he said.

Explaining is one option, but Lo Giudice thinks simulation is also going to play a big role in the next few years as AI-powered tools become mainstream. "Maybe some of the companies that do provide an AI platform are going to provide testing tools for their own products," he said. "Or we might see traditional testing companies jump into that space. This is quite a remarkable problem and opportunity."

Fast development demands fast testing

Jason Arbon knows all too well about the Sisyphean nature of testing, having spent much of his past career working as a tester. "Even before Agile, we really couldn't keep up with the testing requirements," said Arbon, co-founder and CEO of, an AI-powered testing service. While working at Google, Arbon had AI bots doing log monitoring overnight; suddenly, a light went off. Train one group of bots to type into a search box over and over, another to scroll and another to click page links, and suddenly an army is thoroughly testing an application. He won't claim his bots are better than a human tester, but at a single specific task, like refreshing a page, they are more efficient. And there are simply more of them. A company can run tens of thousands of regression tests using AI in the same time a human team could do fewer than 100, Arbon said.

The ability to do more with less was what happened at USA Today, Sypolt said. The newspaper publishes 110 demographic versions daily across a dozen devices, making for a mind-boggling 588 tests that must be run before release. Sypolt brought in the Applitools platform, which offers a visual AI in software testing service, to solve a nagging problem just on iOS devices.

If I freed myself from all technical constraints, my two-year vision would be an AI that has studied all types of users and can act like them.
Torsten Volk managing research director, Enterprise Management Associates

Once that was fixed, Sypolt realized AI could play a much bigger role. USA Today relied on testers to sign off on the UI and UX before production, but they aren't design experts. And the design experts aren't necessarily technical. Sypolt needed a way to bring the design team in at the very end to flag any issues, but he also wanted to feed real-time information back to the designers and testers. AI in software testing turned out to be the bridge that could bring the two sides together and dramatically speed up the development process.

Before AI, it could take up to 90 minutes to run the 588 tests. Once the AI was worked into the continuous integration pipeline, tests finish in 10 minutes. "This has been huge for us," Sypolt said.

But it doesn't stop there. As with many companies, USA Today's development team moves quickly. Workers will push code that might look fine to the naked eye, but AI-enhanced tests uncover problems. "The AI piece is a solid baseline that helps our designers find where issues are located," Sypolt said. "They're not technical, so the AI recognition speeds up their process and makes it easier to communicate to the developers. The whole AI engine has just really sped up our process."

Smarter software testing

As USA Today continues its "digital renovation," Sypolt hopes more AI is in the company's future. Two more visual testing projects -- the newspaper's many videos and new virtual reality "stories" -- are on the top of his list. But he also thinks Swagger's API docs have great potential for AI. "You could generate tests on the fly every time a Swagger doc is generated," he said.

He's not the only one mulling what AI in software testing might bring.

"If I freed myself from all technical constraints, my two-year vision would be an AI that has studied all types of users and can act like them," said Torsten Volk, a managing research director at Enterprise Management Associates. "The AI could have a user persona of a 10-year-old, or a 65-year-old, or someone who is visually impaired."

Mabl's Azeri agreed. "I want to get to the point where I'm not telling [the AI] what to test, but instead it's telling me, 'This is what users are doing with your app.' So these are the tests I should write and then run," Azeri said. "That's our vision. We believe enterprises should be able to basically train Mabl without manually doing anything."

Article 1 of 6

Dig Deeper on Software testing tools and techniques

Cloud Computing
App Architecture