To measure success for AI projects, organizations should properly establish KPIs to both improve the projects' efficiency and enable them to make society better.
Classic AI implementation, for starters, involves machine learning to establish basic models and algorithms and then architect methods of training. After the training process, developers measure their training data against their predicted results to make modifications and reduce errors over time.
There are different types of models, but in all models, it is important to have a measurable output against a known input for training to be effective. Choosing the right tools for the job is equally important, with elements that replicate real world scenarios as closely as possible.
Defining AI KPIs
One of the key metrics in machine learning is mean squared error (MSE). Machine learning courses often introduce this formula as part of their curricula. The nice aspect of MSE is that it exaggerates the impact of outlier results, which can lead to rapid early increases in machine efficiency. The only drawbacks include it being less effective at reducing small errors or measuring effectiveness over repeated learning iterations.
Existing KPIs that have business and IT relevance also apply to AI projects. Typical AI-related KPIs include mean time to repair (MTTR), or how long it takes to fix a problem, and first contact resolution rate (FCRR), which indicates what percentage of problems are resolved by level 1 IT support (basic support) without needing escalation. Additionally, the sheer number of tickets an IT team receives per month is a tangible metric.
Indirect metrics, which tend to be derived from more direct metrics, include customer satisfaction, net promoter scores and total cost of ownership. While indirect metrics are important, it is essential that direct, observable metrics are the basis for any of these secondary metrics.
How KPIs measure AI success
AI-related KPIs help companies measure AI success by ultimately demonstrating a concrete return on investment (ROI). ROI can be expressed as time, money or labor. The best practice would be to use the metric most directly observable and measurable and then translate it to any other metric as necessary.
ROI is usually measured as time -- more specifically, how long it would take to make the same amount of money invested in an AI initiative. Envision an organization that invests $200,000 into an AI initiative and then measured a 20% reduction in MTTR, for example. Since it has 20 staff members using the tool with a combined loaded labor rate of $2 million per year, this 20% cost reduction would come to $400,000 per year. Since it took half of that time (6 months) to make $200,000, that is the ROI.
Another good example is a managed service company. At a typical managed service company, the FCRR is around 65%. Specifically, for every 100 calls coming in, 65 of those calls are resolved by the person who initially took the call, who is a lower-cost first level engineer. This indicates that the other 35 calls would need to go to a specialist for resolution.
This situation adds both time and cost to the resolution, because the second-level engineers that have to tackle these calls have significantly higher salaries. Companies that implement AI to assist the first-level engineers in handling more calls can increase first-contact resolution rates upwards of 80%, reducing both MTTR and FCRR.
AI success should be success for everyone. First, businesses measure success directly, with metrics that are both observable and measurable. Then, they measure qualitative benefits indirectly after determining some directly measurable KPIs. While there are technical metrics aligned with AI, such as mean-squared error, classic IT and business metrics, including MTTR, FCRR and cost per IT troubleshooting ticket, are usually more relatable.