AI shows promise in clinical reasoning, but human oversight remains critical

What is clinical reasoning, and can AI do it?

Clinical reasoning is a key component of a clinician's skillset. However, defining it is challenging, especially across clinical areas. Broadly, clinical reasoning is the process by which clinicians combine their medical knowledge with patient information to identify and understand a patient's diagnosis and determine their treatment plan.

"We throw around the word clinical reasoning and it kind of implies that it's one thing, but it's not," said Nigam Shah, Ph.D., chief data scientist for Stanford Health Care. "There are 183 or so medical specialties, and each one reasons a little bit differently about the problem at hand."

Not only that, but clinical reasoning within a specialty can differ depending on the case and patient. Girish N. Nadkarni, M.D., chair of the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai, noted that clinical reasoning varies from case to case. For example, an emergency medicine physician requires different clinical reasoning capabilities for triage than for diagnosis. Clinical reasoning requirements also vary significantly by patient preferences and patient archetypes.

This variation makes it difficult to measure AI's clinical reasoning competency. According to David M. Liebovitz, M.D., co-director of Northwestern Medicine's Institute for Augmented Intelligence in Medicine, there are methods to measure clinical reasoning; however, there is no agreed-upon standard.

Different methods use different criteria to define clinical reasoning, he noted. While some focus on assessing whether the right data is being examined and whether each data element is being given appropriate consideration, others place greater emphasis on reducing diagnostic uncertainty and examining whether the right types of questions are being asked.

"We have some methods," he said. "They're probably not the best methods, and reasoning alone isn't the whole job of a doctor or caring for patients, but the ways in which we attempt to measure that may not always replicate real-world reasoning."

Publicly available research is also mixed. While some research shows these tools tend to struggle, other research shows they can outperform clinicians on certain clinical reasoning tasks.

The latter study, released at the end of April, compared the clinical reasoning capabilities of advanced LLMs with the baseline performance of human physicians. The study was limited to six clinical reasoning tasks, explained study co-first author, Thomas Buckley, a PhD student in the inaugural Harvard Medical School AI in Medicine Program.

The research showed that the OpenAI o1 series outperformed physicians across tasks, including evaluations of new emergency room patients and clinical management planning.

However, Buckley noted that a wide variety of protocols and data inform the clinical reasoning of a human physician. For instance, an internal medicine physician may need to conduct a physical exam or examine medical images to make clinical decisions, and some of these tasks are still hard -- or impossible -- for AI to do.

"I think we're still very far from completely reproducing all of the aspects of clinical reasoning that a physician does," Buckley said.

Still, the models' performance on clinical reasoning tasks "is pretty astronomical," Buckley said.

"What we're trying to say with this piece is here's evidence that these models demonstrate exceptional clinical reasoning on every test we had essentially," he continued. "Let's start funding clinical trials. Let's start doing these bigger studies of what these things can actually do in clinical care."

Testing will be essential as AI development continues at a frenetic pace. According to Nadkarni, even trying to predict where AI's boundary might eventually lie with respect to clinical reasoning may be a fool's errand.

"What we thought was beyond AI architecture has been changing on a day-to-day basis," he said.

Liebovitz further noted that the latest AI models can not only store and analyze information, but they can also iterate in cycles and potentially solve multi-step problems.

"They've evolved significantly, including things like the repeated iterations that are now possible with some of the reasoning models, where it appears to be that they're thinking out loud almost," he said.

Considering these advancements in healthcare AI, Shah stated that multi-agent AI systems with orchestration layers could feasibly approach the breadth of clinician reasoning that physicians perform.

"If you ask me the question, can a single model ever get to the level of a physician? I would say, I probably don't think so," Shah said. "But if you ask me the question, can a system -- which has a suite of models and an orchestration layer around it and a harness -- can it emulate a physician? Probably very soon, I would say two, three years."

Achieving optimal human-AI collaboration is critical

Despite AI's already impressive advancements and its ongoing march toward clinical reasoning, the technology remains hampered by several limitations.

First, there are the oft-pointed out drawbacks, like hallucinations, inaccuracies and bias, Liebovitz said. In the context of clinical reasoning, these cons could result in AI performing well only in mainstream cases while misinterpreting data from more vulnerable groups or even missing red flags.

"There are blind spots, where training has prevented the [AI's] ability to see past some of those important signals that a clinician in the clinical case would be able to readily pick up," he said.

Second, there is the problem of AI-related de-skilling. Liebovitz noted that with the growing use of AI, there is a corresponding risk that trainees may not learn certain critical clinical reasoning skills and that experienced physicians may even begin to perform poorly if they become too reliant on AI.  

Third, Liebovitz highlighted that AI is still unable to gather and assess certain verbal cues, like tone, and non-verbal cues, such as body language.

If these gaps cannot be sufficiently addressed, AI should perhaps not be allowed to take on more complex clinical reasoning tasks, even if the technology's foundational architecture allows it. Experts agreed that the technology should be used to complement, rather than replace, human clinicians.

For Shah, the optimal relationship between AI and clinicians is that of a pathologist and a cell sorter, a laboratory instrument used to physically separate cells based on their physical, biological or chemical characteristics.

"When I went to med school, I would look in a microscope [at cells] on a slide and manually count how many lymphocytes I saw," he said. "It takes 30 minutes to an hour to do one complete blood count [CBC], and it is notoriously error-prone. Today, I put blood in a tube, put the tube in the machine, and it gives me the CBC. And it's better than me, more accurate. That is true human-machine collaboration, when it is taking work off of my plate."

While AI is providing this benefit across various administrative areas, including revenue cycle management and clinical documentation, developers appear fixated on creating clinically focused AI models to replace physicians' clinical reasoning, according to Shah. But they should instead focus on creating AI that can ease the burdens of clinical care.

For instance, Liebovitz shared that Northwestern is exploring the use of AI to identify gaps in a patient's care. AI could feasibly identify tests performed outside the patient's primary healthcare facility for physicians to discuss at the next appointment or check in with chronic disease patients and determine whether they need to come in and see their clinician sooner.

"I do think healthcare is ripe for disruption," Liebovitz said. "I think there are lots of opportunities at population-based levels where we can really help patients."

Not only that, but AI could also help expand the clinical workforce in areas where shortages are limiting access. Buckley pointed to the primary care access crisis in the U.S. as one such area.

"I think we're still in very early stages, but I still think these technologies have the opportunity to, one, reduce the burden of primary care physicians currently, and then also increase access to care for patients if they have some sort of autonomous AI agent that they could see for care initially and then maybe be referred to a physician if they need it," he said.

While there are no technical limitations on AI's current and future ability to perform clinical reasoning tasks, one thing is clear: humans must not only stay in the loop but also be at the helm of AI development as the technology evolves and its challenges are addressed.

Anuja Vaidya has covered the healthcare industry since 2012. She currently covers healthcare IT and innovation, including artificial intelligence, digital healthcare, EHRs and interoperability.