As AI tools multiply, health systems are centralizing AI management

Scaling AI governance without a one-size fits all approach

Before implementing the Signal 1 platform, Inova employed various tools to manage its AI inventory.

The health system used Word documents to collect information from vendors as well as ethics assessments completed by the Inova leader sponsoring the use case. Inova also tracked its AI inventory in spreadsheets and used qualitative tools to monitor safety and reliability. While performance reports from vendors were available, they were specific to individual AI features, making it difficult to compare performance across the organization. As Inova's AI portfolio grew, leaders recognized that these processes would become difficult to sustain.

McManus highlighted that Inova's experience reflected a broader challenge for large health systems as their AI portfolios expand.

"Volume [of AI tools is] going to require a platform approach for enterprise-size companies," he said.

Mount Sinai also faced AI governance scaling challenges despite already having a centralized AI governance structure. As part of that structure, the health system has an independent AI assurance lab -- a dedicated team that evaluates higher-risk AI tools before deployment and monitors them once they are in use. The assurance lab tests tools for performance, bias and potential unintended consequences or harm before making implementation recommendations to the health system's AI governance team.

However, applying that assurance process across Mount Sinai's AI portfolio could be labor-intensive. For scaling individual AI tools, staff may need to map workflows, design evaluations, configure and run tests and establish ongoing monitoring, according to Freeman.

So, Mount Sinai is testing whether the Signal 1 platform can reduce the work required to conduct assurance across its AI portfolio.

"We can't yet automate that full process, but we can shorten the time, energy, effort it takes to do assurance on each one of those AI tools that require it," Freeman said. 

Implementing an enterprise AI management platform

Inova's implementation of the Signal 1 system involved a great deal more than deploying an off-the-shelf platform. As a co-development partner, the health system had worked with Signal 1 for about 10 months at the time of the interview to develop governance workflows, automation and approaches for measuring AI safety and value.

When standing up the platform's monitoring module, for instance, Inova began onboarding individual AI applications. The process initially required Inova teams to determine which data connections and performance measures were appropriate for each AI use case -- including tools for identifying sepsis, voice-based workflows, automated patient message responses and computer vision technologies.

But, establishing those measures individually made it time-consuming to bring new AI applications into the monitoring process.

"We were seeing a time to market that was lengthy and really reinventing the wheel each time," McManus said.

In response, Inova and Signal 1 developed baseline sets of measures that McManus said cover about 95% of what is typically needed to monitor an AI application. Signal 1 said via email that the measures vary by type of AI, and cover performance, fairness and safety when relevant.

For example, a model designed to classify something into one of two categories might be evaluated on how accurately it distinguishes between those categories, while a generative AI tool might be assessed on whether its output is complete and supported by the source information, Signal 1 said. Teams can then add the measures specific to an individual use case rather than building their monitoring framework from scratch.

"We're not losing any integrity because people are still making the fine-tune adjustments, but they're doing it on top of 95% of the work done instead of working with a blank palette and trying to build it all out each time," McManus said.

According to McManus, adding AI tools to the monitoring module took roughly six to eight weeks while the broader co-development work continued.

Mount Sinai's implementation is at an earlier stage. At the time of the interview, the health system was about six weeks into the project and had brought its first few use cases live on the management platform. Signal 1 is being incorporated into the health system's existing assurance workflow, with the AI assurance lab team beginning to use the platform.

The team will start with four substantially different technologies, including a generative AI feature to help with in-basket messages, an internally developed predictive model and a medical imaging application. The health system will evaluate whether Signal 1 can match or exceed its existing assurance processes while reducing the staff time and resources required before deciding how broadly to expand the platform.

"You have to have a really robust evaluation kind of capability to just know how these tools are working with your patient population," Freeman said.

Both health system leaders hope that a more centralized approach to AI management will help governance keep pace with their growing portfolios without sacrificing rigorous evaluation and monitoring.  

Elizabeth Stricker, BSN, RN, comes from a nursing and healthcare leadership background, and covers health technology and leadership trends for B2B audiences.