tippapatt - stock.adobe.com
How a clinical-reasoning framework can help improve smaller AI models' performance
Researchers built a clinical framework to study the reasoning of smaller open-source AI models to make them more accessible and cost-effective, especially in rural areas.
As health systems work on reducing patient readmissions, clinicians can benefit from affordable and accessible AI models to help them make key decisions, like the right time to discharge patients. However, small and rural hospitals are often unable to afford expensive large language models that provide more reliable recommendations to inform clinician decision-making.
That's why researchers at Carnegie Mellon University, Weill Cornell Medicine and the Indiana University School of Medicine have created a new framework called Differential Reasoning Learning to examine discrepancies in the performance of smaller AI models or clinical agents, compared with larger models, and thereby improve them. The study also highlights how the smaller open-source AI models are more affordable and accessible than larger ones.
"This project demonstrates the potential that this class of models has in settings where there is a need to balance high-quality performance and cost factors," said Ramayya Krishnan, Ph.D., professor of management science and information systems at Carnegie Mellon University's Heinz College and College of Engineering.
To improve clinical decision-making, the DRL framework not only compares smaller AI models to larger ones, but also the performance of these agents against physician-authored clinical reasoning and clinical guidelines.
"The objective is how do we get small models to be good at making these decisions because we want them to be more widely deployable," Krishnan said. "But to do that, we need to have the capacity to compare them to how large models perform, and then the gold standard is how human physicians perform in these consequential tasks."
LLMs comprise frontier models such as Claude Fable 5, Claude Opus 4.8 or GPT-5.6, Krishnan said. The smaller types consist of Microsoft Phi small language models or Qwen, a series of Chinese open-weight models from Alibaba.
A multidisciplinary group of computer scientists, as well as healthcare policy and AI experts, conducted the study, which was published on AirXiv, an online repository for scientific research papers.
What researchers learned about small AI models
A key goal in creating the framework was to evaluate the quality of an AI tool's clinical recommendation and how it came to its conclusion. The evaluation focused on the reasoning trace, or chain of thought, of small AI models. In the framework, a series of graphs represents the reasoning trace.
"Every decision has both the decision-making quality and reasoning trace," Krishnan said. "We're interested in both of those."
Researchers examined the reasoning traces of small models for errors of omission and commission. Omissions are questions small models failed to ask of patient data that large models did ask, and commissions are concepts that small models considered regarding patient data that they should not have, Krishnan explained.
Researchers discovered that smaller models trail large models in performance because they hallucinate certain concepts. Then, as they applied the framework to emergency departments, they found that omitting key data can affect clinical decision-making, including follow-up plans and patient outcomes, Krishnan noted.
However, researchers also found that they can edit the small models' reasoning traces to improve the AI models' performance, reducing missing nodes, hallucinations and path errors in the models, according to the paper.
To boost performance, clinicians can ask questions that larger models ask that are appropriate to the patient, such as whether the patient is a single, older person and whether they have had a fall, according to Krishnan. Asking these additional questions provided more context for smaller, less capable models, allowing clinicians to train the models to improve their performance on decision-making tasks.
With this extra context that the small models gained following a reasoning trace analysis, they could make better predictions about which patients should be discharged from the ED, Krishnan explained.
For instance, adding the extra questions using the framework could boost the small model's performance to within 86% of a large model's performance, Krishnan said.
Co-author Rema Padman, Ph.D., trustees professor of management science and healthcare informatics at Carnegie Mellon University's Heinz College, compared giving the small models more context to providing hints to a child in a guessing game.
Each hint brings the child closer to the correct answer, and in this case, more context improves the performance of small AI models, she explained.
Krishnan added that the framework incorporates an implementation pipeline that serves as a "testing infrastructure" to get the small models ready for deployment.
"This edit-evaluation pipeline has been implemented as a set of software modules and demonstrated to improve small model performance in the context of emergency room discharge decisions," he explained.
Krishnan further noted that large models don't require this degree of pre-deployment evaluation.
"It's very likely the large models don't need to do this much additional work because they pretty much come ready out of the box, but they're much more expensive," he said.
Benefits of using a clinical reasoning framework to enhance small AI models
Researchers developed the framework to examine how under-resourced hospitals, such as small facilities in rural areas, could avoid the high costs of larger models, Krishnan said. By improving the performance of smaller models, the framework can help make the models that small and medium-sized hospitals use more affordable, widely accessible and easily deployable.
Padman similarly explained that the study aims to show that smaller language models can bring multimodal data together in a low-cost manner for hospitals in different regions, including rural, urban and semi-urban areas.
"Every hospital is trying to come up with its own standard machine learning-based prediction model, but there is just so much information available, not just in the EHR, but also in the progress notes on the patient," Padman said.
When deciding whether to discharge a patient, ED clinicians want to avoid discharging too early so they don't end up back in the ED. Therefore, researchers compared the reasoning of small models to large ones as well as clinicians on when they would discharge a patient and how this timing affects when a patient may need to return to the ED, Krishnan said.
"Patient readmission is such a consequential issue both from the perspective of quality of care as well as costs of care, so ways in which we can leverage AI and these foundation models at the lowest cost possible -- which is why we are thinking about using small models -- are an advantage," Padman said.
AI models are designed to augment, not replace, physician decision-making
Rohan Ramakrishna, M.D., a brain tumor neurosurgeon for Weill Cornell Medicine and cofounder of Roon, an online physician community, sees real value in the Differential Reasoning Learning framework, noting that "the architecture is a real contribution: auditable, editable clinical knowledge you can inspect."
He further suggested that by analyzing the performance of AI models, the framework allows physicians to evaluate their own reasoning.
"By investigating the model, we also investigate ourselves and hopefully uplevel in the process," Ramakrishna said. "We also learn where the models break down, so that when we use models, we have an internal sense of how they might fail."
Krishnan noted that while the framework sheds light on how AI models aid ED discharge decisions, physicians must still make the final call.
"The AI recommendation should be integrated into the workflow in a way that the physician can use it and choose to either agree with what the AI is recommending or disagree with it," Krishnan said. "But in effect, it's designed to augment the physician; it's not designed to replace a physician."
Padman also sees value in clinicians' understanding why these AI models make certain recommendations as they use the tools to make their own decisions on hospital discharges. AI models provide context about a patient's overall health and recommendations about next steps, but then the physicians themselves must decide when the patient should be discharged from the hospital, according to Padman.
She predicted that the Differential Reasoning Learning framework will be incorporated into other clinical-reasoning frameworks that clinicians already use.
Ultimately, by improving how small language models work, researchers hope that AI models will become more accessible.
"If we can build powerful small language models that come close to the performance capabilities of large language models by figuring out each clinical context -- ED, oncology, cardiology, and so on -- the additional information will really enhance the power of these small language models," Padman said. "I think that is particularly of value in healthcare."
Brian T. Horowitz started covering health IT news in 2010 and the tech beat overall in 1996.
