Addressing concerns regarding health AI use in 2026

A lack of clinical oversight

A key concern around AI in healthcare is the lack of proper clinical oversight, noted Jonathan Kron, CEO of BloodGPT, a platform that uses AI to interpret blood test results. Kron says AI can be "persuasive," even when it's wrong, so having clinicians oversee the work of AI is essential.

"Enabling it to act without the approval or steering of a clinician increases risk," Kron said.

The fast pace of large language model (LLM) development also creates challenges for clinicians looking to establish oversight over AI, explained Holly Wiberg, Ph.D., assistant professor of operations research and public policy in the Heinz College of Information Systems and Public Policy at Carnegie Mellon University.

Rather than evaluating AI tools one time, they should be part of a more flexible and ongoing monitoring strategy, especially given the dynamic nature of genAI tools, according to Wiberg.

"LLM-based systems are subject to frequent model revisions, necessitating ongoing monitoring to ensure stability, safety and regulatory compliance," Wiberg said.

She recommended that health systems automate recurring monitoring of model performance.

Atreja also noted concerns around a lack of structure to monitor AI algorithms. To create this structure, Atreja recommended evaluating if models are private, secure, ethical and compliant.

Additionally, determine if the model is generalizable for a population, he advised. Generalizable AI models perform accurately and reliably across varying groups and settings.

"But after implementation, the models do drift based on the population, based on how models evolve," Atreja explained. "And so, you need continuous monitoring, and most of the organizations don't have a way to continuously monitor these models."

By monitoring AI algorithms continuously, clinicians can prevent safety risks arising from model accuracy decreasing as populations change, he suggested.

"That can cause wrong decisions to be made potentially," Atreja said.

He also highlighted the potential lack of accuracy of AI scribes due to missing relevant data.

"People have also seen with AI scribes, many times you have to correct what is written, and if you just ignore and you don't correct, then there's a chance wrong information may carry over in the note," Atreja said.

In addition, a lack of vetted information brings challenges as patients are adopting AI tools faster than health systems. Wiberg noted that patients are increasingly relying on tools like ChatGPT to research symptoms and seek medical advice.

"The relatively slow health system adoption and contrasting rapid patient adoption create a gap where patients seek health information," she said. "Health systems must grapple with how to balance well-placed organizational caution with the opportunity to offer vetted information to meet patient needs."

Data rights and transparency

Blindly incorporating data into AI tools can create problems with data rights and a lack of transparency, according to Kron.

He advised that tech vendors use properly de-identified aggregate data to ensure that AI systems are safe and fair.

He also said companies should not use identifiable protected health information (PHI) in AI tools unless they have received "explicit consent" from the patient and have made the purpose clear.

In addition, Kron said organizations should not retain PHI in production, and "governed de-identified learning" data should deliver insights and value for the original data owner. He recommended clear rules and short retention as well as conducting audits.

Further, Kron recommended "nutrition labels" to maintain transparency on how AI tools are deployed.

"If a vendor can't show a simple 'nutrition label' illustrating what data they use, whether PHI is retained (it shouldn't be), how de-identification is done, when the model last changed, known failure modes and basic equity checks, we should treat it as not ready for care," Kron said. "No label, no deployment."

Too many AI models in healthcare

The flood of AI models, agents and solutions in healthcare is causing a "problem of plenty," according to Atreja.

"It's creating a very fragmented marketplace, and that's causing decision paralysis," he said. "People don't know which one to choose and double down on."

Atreja said that what companies such as AWS, Google, Microsoft and OpenAI offer are "general-purpose solutions" that are not "purpose-fit" for healthcare or integrated with EHR records or data.

"There's a lot of work that needs to be done from what is being built or made available, in general, in AI to what we need in a trustworthy fashion within the health system, with our workflow, to really create value from it," Atreja said.

"You can imagine, people will have tens to hundreds of AI algorithms working in the organization, and you don't want to log in to each one of those applications separately and monitor," he continued. "You want to have a centralized system to do it, and that's a big unmet gap."

Fitting into healthcare workflows

A plethora of AI assistants can lead to fragmentation in workflows, and the healthcare industry is leery of AI tools that don't fit seamlessly into everyday routines, according to Kron.

"Clinicians don't want to bounce across apps," Kron said. "They want fewer steps to get to the next action."

To fix this problem, Kron recommended choosing AI tools that fit inside a health system's workflow and can enable actions such as ordering, scheduling and coverage checks with clinician oversight.

"If AI sits outside the EHR/lab/payer rails, it adds clicks and errors," Kron said.

Additionally, complex, unstructured notes consisting of raw audio from patient visits complicate integration of tools such as AI scribes into clinical workflows, Wiberg explained.

Completed notes from patient visits often consist of "free-text, unstructured outputs," she added.

This unstructured data gets fed into AI models, and the raw audio and free-flowing text can make developing evaluation metrics challenging within complex workflows because they involve many stakeholders, according to Wiberg.

"Model outputs rarely stand alone," Wiberg said. "Instead, they are used across a range of downstream tasks, from claim preparation and medication reconciliation to serving as a reference in future visits by the original provider or referred specialists."

As AI model outputs are reused and repurposed in complex systems, structured evaluation becomes difficult for health systems.

"We care not only about how the model performs in isolation, but also about tracing its downstream impact," Wiberg said.

As health systems integrate AI tools into their workflows, they must also decide whether to develop them internally or use products from external vendors, according to Wiberg. Health systems that choose to develop AI tools in-house must understand that it will require significant investments in staff time, training and monitoring as well as the financial costs of training models.

"Fine-tuning models can be quite expensive," she said. "These high-resource needs make third-party tool acquisition quite appealing as an alternative."

"However, third-party tools introduce new security and oversight vulnerabilities," she continued. "As stewards of patient data, health systems must ensure that vendors use their data responsibly, including that the data are not used to train commercial LLMs."

Preparing for safe, effective healthcare AI use in 2026 

To improve trust in AI apps with models and plugins changing fast, health systems should create an "approved uses/vendors" list, Kron said. This list would be part of an AI formulary that outlines which use cases are allowed and where AI tools can be run.

"Keep it short and update it regularly," Kron suggested.

To achieve safe AI use in healthcare, he also recommended keeping automation supervised.

"That means nothing changes in the record or triggers care without clinician approval," Kron said.

Further, organizations should train not only clinical faculty on how to use AI responsibly but also the C-suite on the best use cases in which health AI can create an impact, according to Atreja.

Atreja recommends a program called AIM-AHEAD All of Us for training researchers in AI and machine learning (ML). Meanwhile, the American Board of Artificial Intelligence in Medicine (ABAIM) offers medical AI education for clinicians pursuing board certification.

Although AI programs may be becoming more efficient, health systems shouldn't let their guard down as far as trust and supervision go to ensure safe and effective AI use in 2026.

"We should not confuse intuitiveness and how good they look or how efficient they are with accuracy," Atreja said. "So we need to maintain our guard in terms of accuracy and trust and have supervision so they don't make errors."

"As long as we do use them with discretion and supervision, I think we can create value," he added.

Brian T. Horowitz started covering health IT news in 2010 and the tech beat overall in 1996.