This content is part of the Conference Coverage: 2018 MIT Sloan CIO Symposium: A SearchCIO guide

With artificial intelligence and machine learning, KPIs are crucial

The CIO of DBS Bank and CTO of Kayak square off at the 2018 MIT Sloan CIO Symposium on how to use AI for business success.

Two technology leaders who appeared on stage at the recent MIT Sloan CIO Symposium represented opposite ends of the AI proficiency spectrum.

At one end was Kayak, a digitally native company founded in 2003 that has seamlessly integrated artificial intelligence and machine learning into its product development lifecycle. At the other end was DBS Bank, a multinational financial services company established in 1968 by the Government of Singapore.

The two companies may not be competitors, but with their technology leaders sitting side by side on stage, it seemed only natural to wonder: Can a DBS Bank ever catch up to a Kayak when it comes to implementing artificial intelligence? The answer seems to be yes, but not without drive, vision and well-thought-out key performance indicators (KPIs).

Building an AI-conversant enterprise

DBS Bank is in the midst of an enormous digital transformation effort, one that includes an aggressive plan to implement artificial intelligence and machine learning. Part of the near-term effort is to build an enterprise AI competency.

"We want to train the top 250 executives in the firm to become AI conversant," said David Gledhill, group CIO and head of group technology and operations at DBS Bank. "We want to train 200 people to be what we call AI translators."

By conversant, Gledhill means training executives to understand machine learning techniques at a high level, as well as knowing enough to "intelligently question what a model is and what it can do," he said.

DBS Bank, AI, implementing AI, implementing artificial intelligence, MIT CIO Symposium, CIODavid Gledhill

The numbers he shared are more than a want, they are key performance indicators for the year, and they are tangible enough to give employees a realistic goal without drowning them in the vision thing. "We don't want to go too far, too soon because we think that's just going to completely alienate people," Gledhill said.

And, ultimately, this executive cadre is an important part of a larger strategy to grow the company and to foster a reputation as a premier digital bank. Gledhill described the education efforts as "phase one" of a multiphase plan. Phase two metrics are being hashed out now but will focus on "how to solve process at scale." That could include KPIs that measure process automation as well as targets for company growth.

Phase three is still a work in progress but will likely focus on observation and prediction. "A goal that one of our teams there has already set is how do I detect a million problems before they happen," Gledhill said.

The customer comes first

Gledhill said he sees artificial intelligence and machine learning as "just another tool in the toolbox" and the implementation of the technologies as an extension of the data culture already in place.

What's missing is employee training and a supporting workflow design. When Gledhill and his team began asking what it means to design for AI, they helped guide the conversation to focus on "data capture, data governance, instrumentation, and also the way you design a process and what is applicable to AI," he said. "Training people to understand what that actually means is kind of the challenge and the opportunity."

At Kayak, artificial intelligence and machine learning are also seen as tools in the toolbox, but KPIs that measure their usage simply don't exist. The company, which aggregates prices on flights, hotels and car rentals, has been using artificial intelligence and machine learning behind the scenes for years.

Kayak, AI, implementing AI, implementing artificial intelligence, MIT CIO Symposium, CTOGiorgos Zacharia

"At the end of the day, it's about measuring user experience," said Giorgos Zacharia, Kayak CTO and chief scientist who holds four math and computer science degrees from MIT. "If machine learning is the right tool, you use it. If it provides improvement, you use it."

When pressed to share UX KPIs that might influence developers to choose artificial intelligence and machine learning tools over traditional coding techniques, Zacharia insisted the company just doesn't think that way. "We basically have one KPI: Have we helped users find the flight, hotel or car they were looking for?" he said. "Whatever tools you use underneath, they have to drive toward an improvement for that."

Indeed, customer experience appears to influence just about every decision at Kayak. Even when talking about security measures, latency and fraud detection, Zacharia brought his point back to user experience. "Our own suppliers of information -- airlines, online travel agencies, et cetera -- love our data," he said. "Some of them actually hire bot scrapers to find out their competitors' prices. So protecting the website from malicious -- and non-malicious -- bots is very important because it affects user experience."

A shared goal

The distinction between the detailed, multiyear plan from Gledhill and the laser-like focus on customer experience from Zacharia drove Michael Schrage, MIT researcher and panel moderator, to pit the two against each other. "What advice would you offer [Gledhill]?" he asked Zacharia. "Do you think he's being too process-oriented on this?"

"Maybe. But it's a bank," said Zacharia, drawing a hearty laugh from the audience.

Gledhill didn't let the comment slide, asserting that the financial institution's goal to solve a million problems before they happen is directly tied to customer experience. He described DBS Bank as being "on this journey" to make customer experience less of a passive endeavor and more of a predictive one.

"As we evolve our product set," he said, "we've kind of gone on this journey of customer experience from customer feedback, which simply doesn't work out well, to design thinking to what we're now exploring, which is this whole area of customer science," he said. He said it was important to understand the science of customer behavior, to find areas where customers struggle and to determine what great service looks like.

We are at war, and, as my boss keeps reminding me, digital and technology is the way we're going to win or lose.
David Gledhillgroup CIO, DBS Bank

Rather than designing for AI, DBS Bank's developers will eventually design for what Gledhill called "customer ops." That includes finding the friction points customers might experience, figuring out how long a customer is likely to wait for a response before moving on to the next transaction, aligning KPIs to a customer ops dashboard in an effort to stay one step ahead of customers, and using cutting-edge technologies like artificial intelligence and machine learning to make it happen.

"We are at war, and, as my boss keeps reminding me, digital and technology is the way we're going to win or lose," Gledhill said.

Dig Deeper on Digital transformation

Cloud Computing
Mobile Computing
Data Center
and ESG