How to ensure interpretability in machine learning models

What is interpretability in machine learning?

In machine learning, interpretability is the ability to understand why a model makes a given decision.

For example, consider a simple model that sorts images of cats from images of dogs. If the model is highly interpretable, users can understand the categorization process that takes place in the model when it assesses each image. Users can discern the parameters the model considers and how it processes them to arrive at a conclusion. Does it emphasize overall colors in each image? Or does it look for certain color concentrations, like black dots that could indicate the presence of a dog's snout? With interpretability, users can answer questions like these to some extent.

Importantly, interpretability exists on a spectrum; some models are more interpretable than others. Even with highly interpretable models, it's often impossible to know with certainty why a model makes a decision, but it's possible to gain a fairly thorough understanding of the process.

Why does interpretability matter?

Model interpretability is important in several areas, including compliance, transparency and managing AI system behavior.

Compliance and transparency

Some compliance regulations require that businesses explain to users why an automated service makes certain decisions about them. For instance, the EU's GDPR mandates transparency in algorithmic decision-making, and the EU Artificial Intelligence Act requires organizations to be able to explain how they made certain types of decisions when those decisions involve AI.

The more interpretable a model, the easier it is for a company to customize and control AI behavior.

More generally, users might demand transparency, even if no compliance rule requires it. For example, a consumer whose loan application is denied by an AI service might want to understand the reasons behind the decision so they can rectify the issue and reapply.

In cases like these, model interpretability is critical. Without it, models remain black boxes where decisions are opaque even to the companies that deploy them. Moreover, ML bias is more likely to go undetected and harm users when a model lacks interpretability.

Managing model behavior

The more interpretable a model, the easier it is for a company to customize and control AI behavior. Users that have information about the parameters a model evaluates and how it assesses them are better positioned to modify those parameters and influence the model's operations.

For example, a bank might have to fix an AI tool that generates a financial report containing inaccurate data. If the model underlying the tool is highly interpretable, the bank might discover the tool didn't have access to a critical database. Consequently, it hallucinated data instead of drawing on actual data. The solution in this case would be to connect the tool to the database to stop the hallucinations and improve its performance. Without interpretability, the bank wouldn't be able to trace the AI decision-making process that generated inaccurate output. It would end up shooting in the dark as it sought to remediate the problem.

Interpretability vs. explainability vs. auditability

ML interpretability is related to two similar concepts -- explainability and auditability. But these are distinct practices, and it's important not to conflate them with one another.

Explainability

Explainability refers to the ability to predict which types of decisions a model will make based on specific types of input. For instance, if users know a model mislabels yellow-coated cats as dogs -- even if they don't know precisely why it mislabels them -- the model is explainable because users can anticipate that certain inputs will yield certain outputs.

Unlike interpretability, explainability doesn't require insight into a model's internal workings. It merely involves predicting how a model is likely to respond to given types of input based on observed patterns. Users can establish explainability by looking at examples of past predictions and comparing input to output.

Explainability is useful for anticipating problematic model behavior and can help users tweak input data to control model output. However, it doesn't aid in customizing or troubleshooting the model itself, as it provides no insight into a model's inner workings.

Auditability

Auditability, like interpretability, focuses on understanding how models generate certain results, but it's more technical and specific. Auditability typically aims to reconstruct the exact decision-making path that a model works through and the specific data it considers when making a decision. In contrast, interpretability is about understanding a model's general decision-making strategy.

An auditable AI tool might analyze photo footage to determine whether cars have run a red light. To charge a particular driver with running a light, the model would need to be able to identify the specific parameters it considered, such as the car's license plate, the time the vehicle entered the intersection and when the light changed. In other words, the system would have to be able to prove exactly how it reached a conclusion about a specific case.

For that same system to be interpretable, it would require simply understanding how the model operates overall, not identifying the exact parameters that led it to make a certain decision.

Auditability is important for use cases that involve decision-making in highly regulated medical, legal or financial contexts. These situations often require the ability to document, reconstruct or repeat AI-driven processes.

Interpretability in the age of generative and agentic AI

Historically, AI systems tended to have relatively high degrees of interpretability for several reasons, including the following:

• The systems were generally less complex in design, making it easier to trace how they operated.
• They ingested less data, which meant there were fewer parameters to consider when interpreting a model's behavior.
• Older AI systems were often domain specific. They were designed for a specific use case, such as making product recommendations or converting speech to text. This made it easier for developers to ensure interpretability because the models worked within a narrow context.

Agentic AI … makes the interpretability process even more complex because AI agents might interact with one another or multiple models.

Much has changed since generative and agentic AI applications have emerged. These systems are powered by extremely complex models trained on trillions of parameters. The models often are general-purpose, or generic, designed to process any type of information. And even if an AI tool serves a narrow use case, it might be powered by a generic model. For instance, the same large language models behind software development tools might also drive applications that create financial reports and marketing content.

Agentic AI combines software tools with AI models to automate actions, making the interpretability process even more complex because AI agents might interact with one another or multiple models.

Nonetheless, generative and agentic AI play a critical role in business environments. That, combined with regulatory scrutiny to ensure fairness and transparency in AI-assisted processes, means that companies must make their AI systems as interpretable as possible, even if the underlying technology doesn't make it easy.

Strategies for ensuring interpretability in ML models

The best way to achieve interpretability in AI systems depends largely on whether an organization develops its AI models from scratch or relies on models supplied by a third-party vendor.

Building interpretability into models the business controls

When designing a new model from scratch, the easiest way to ensure interpretability is to use an algorithm that operates in straightforward, predictable ways. Following are some common examples of such algorithms:

• Linear regression. A type of algorithm that makes decisions by comparing unknown data to known data.
• Generalized linear modeling. A method that focuses on probability distributions across non-normal distributions of data.
• Decision trees. An approach that works through a series of questions to arrive at a conclusion.

Models based on these algorithms offer an inherent degree of interpretability, because the algorithms behave in understandable ways. This approach contrasts with methods like deep learning, which involves multiple layers of processing. Predicting how a deep learning model will process a given layer -- let alone how it will behave across multiple layers -- is challenging and leads to low interpretability.

Interpretability for third-party models

When working with third-party models, ensuring interpretability can be a challenge because they're black boxes. Businesses typically have no control over how external models work or report data. Interpretability data might only be available to the AI vendor but not shared with customers, who can only view model outputs.

Still, the following tactics can help achieve some level of interpretability, even when working with black box models:

• Choose AI tools and vendors that offer interpretability features and assess the depth and reliability of these capabilities.
• When interacting with AI systems, engineer prompts that include instructions to explain the decision-making process.
• Keep prompts and requests simple. Simpler requests, with fewer variables and parameters to consider, are generally easier to interpret.
• To analyze AI system behavior, use post hoc interpretation methods, such as local interpretable model-agnostic explanation, Shapley additive explanations or partial dependence plots. These approaches map model input-output relationships to infer decision-making processes. Though they offer some level of insight, these techniques don't generate true visibility into a model's internal workings.

The future of interpretability in ML models

Improving AI model interpretability will require technical and cultural change.

Technically, AI developers will need to build better methods of achieving visibility into model decision-making without compromising model complexity or scale. No clear-cut solutions exist, but interpretability enhancements remain an active area of AI research.

Culturally, moving the needle on interpretability requires greater prioritization of AI system transparency and visibility. The compliance ecosystem is moving in this direction; AI regulations like the EU AI Act include requirements related to explaining decision-making.

More broadly, AI vendors and their customers will need to decide that ensuring interpretability is at least as important as factors like the speed, accuracy and efficiency of AI models. Interpretability hasn't been a key focus of major AI companies, but that could change if AI buyers demand better interpretability features.

Chris Tozzi is a freelance writer, research adviser, and professor of IT and society who has previously worked as a journalist and Linux systems administrator.