OpenEvidence adds real-time evidence quality grading to AI

How EvidenceGrade works

When a physician asks  OpenEvidence a question and receives an answer, EvidenceGrade first determines whether the question at hand boils down to a clear evidentiary claim. The system filters out simple definition lookups and queries focused on tasks like summarization.

Once the question has been determined to be gradeable, each published paper included in the answer is rated for quality, certainty and relevance and receives a grade from A to D. Grade A means that the evidence used rigorous systematic reviews, whereas grade D could mean that the evidence consists of preclinical or animal data and cannot be relied upon.

In a blog post accompanying the announcement, OpenEvidence explained that users have long been requesting a systematic assessment of the strength of the evidence surfaced in its answers. However, manually grading medical evidence is time-consuming and challenging. As a result, only a fraction of the questions clinicians ask in practice are subject to formal evidence appraisals.

EvidenceGrade is OpenEvidence's attempt to bring the transparency of evidence grading to physician users without sacrificing speed, the blog post explained.

"Not all evidence is equally certain, and when a clinician acts on an answer, it's imperative they understand how much weight the underlying evidence can bear," Samuel Finlayson, M.D., senior vice president of medical AI at OpenEvidence, said in the announcement.

"Expert research appraisal teams do invaluable work, but they can't cover every question that comes up in everyday practice. We built EvidenceGrade to extend their methods to as many questions as possible. We see it as a complement to traditional evidence synthesis, and we're releasing it as a starting point that will improve as the clinical community helps us refine it."

OpenEvidence vs. general-purpose AI tools

OpenEvidence has gained significant popularity since its founding in 2021 and is now used by nearly two-thirds of U.S. physicians. However, a study published in nature medicine in June 2026 found that general-purpose AI tools like GPT-5.2 and Claude Opus 4.6 could outperform specialized LLM-based clinical AI tools such as OpenEvidence and UpToDate Expert AI on multiple medical benchmarks.

The study raised questions about the value of creating proprietary clinical AI tools for specific industries versus using general-purpose tools or developing hospital-specific LLMs.

In a LinkedIn post responding to the research, OpenEvidence argued that the paper and its benchmarks are "clearly ill-equipped to judge clinical decision support tools or draw conclusions about specialized clinical AI," alleging that the authors have an "obvious" conflict of interest and "inattention to basic data contamination concerns."

Days after the nature medicine study was published, OpenEvidence reached out to a UCSF researcher to conduct a new study, STAT reported. OpenEvidence reportedly supplied the data, paid the survey respondents and co-developed the data collection plan.

The preprint results compared OpenEvidence to general-purpose tools and found that "physicians scored the specialized tool highest on all axes," in complete opposition to the nature medicine study.

OpenEvidence did not disclose its involvement in the study, nor are its executives listed as authors. As the AI race continues, specialized platforms like OpenEvidence will likely face increased competition from tech behemoths like OpenAI and Anthropic.

Jill Hughes has covered health tech news since 2021. Her coverage areas include cybersecurity, HIPAA compliance, interoperability, AI and EHRs.