Researchers found that ChatGPT 3.5, a form of artificial intelligence, fared worse than PharmD students in answering questions on therapeutics examinations that ensure students have the knowledge, skills, and critical thinking abilities to provide safe, effective and patient-centered care.

ChatGPT was less likely to correctly answer application-based questions (44%) compared with questions focused on recall of facts (80%). It also was less likely to answer case-based questions correctly (45%) compared with questions that weren’t focused on patient cases (74%). Overall, ChatGPT answered only 51% of the questions correctly.

The results provide additional insights into the uses and limitations of the technology and may also prove valuable in the development of pharmacy exam questions. The study findings appear in Currents in Pharmacy Teaching and Learning.

“AI has many potential uses in health care and education, and it’s not going away,” said Christopher Edwards, PharmD, an associate clinical professor of pharmacy practice and science. “One of the things we were hoping to answer with the study was if students wanted to use AI on an exam, how would they perform? I wanted to have data to show the students and tell them they can do well in the exams by studying hard and they don’t necessarily need these tools.”

A secondary goal was to find out what kinds of questions AI would struggle with. Coit College of Pharmacy Interim Dean Brian Erstad, PharmD, wasn’t surprised that ChatGPT did better with straightforward multiple choice and true-false questions and was less successful with application-based questions.

“The kinds of places where evidence is limited and judgment is required, which is often in a clinical setting, was where we found the technology somewhat lacking,” he said. “Ironically those are the kinds of questions clinicians are always facing.”

Edwards, Erstad, and Bernadette Cornelison, PharmD, an associate professor of pharmacy practice and science, evaluated answers to 210 questions from six exams in two pharmacotherapeutics courses that are part of the university’s Coit College of Pharmacy PharmD program.

The questions came from a first-year PharmD course focused on disorders related to nonprescription medications for heartburn, diarrhea, atopic dermatitis, cold and allergies. The other class was a second-year course that covered cardiology, neurology and critical care topics.

To compare the exam performances of pharmacy students and ChatGPT, they calculated mean composite scores as a measure of the ability to correctly answer questions. For ChatGPT, they added individual scores for each exam and divided by the number of exams. To figure out the mean composite score for the students, they divided the sum of the mean class performance on each exam by the number of exams. The mean composite score for six exams for ChatGPT was 53 compared to 82 for pharmacy students.

Educators, clinicians and others continue to debate the value of AI large language models, such as ChatGPT, in academic medicine. While such models will continue to play a range of roles in health care, pharmacy practice and other areas, many are concerned that relying too much on the technology could hamper the development of needed reasoning and critical thinking skills in students.

Both Erstad and Edwards acknowledged that in time, newer and more advanced technology may change these results.

Image: Brian Erstad, PharmD, is the interim dean and a professor at the R. Ken Coit College of Pharmacy.

View more Credit: Photo by Kris Hanning, U of A Office of Research and Partnerships

These tools, known as large-language models, are already being used by patients to make sense of their symptoms and medical tests results.

But while these AI models perform impressively on standardized medical tests, how well do they fare in situations that more closely mimic the real world?

Not that great, according to the findings of a new study led by researchers at Harvard Medical School and Stanford University.

For their analysis, published Jan. 2 in Nature Medicine, the researchers designed an evaluation framework — or a test — called CRAFT-MD (Conversational Reasoning Assessment Framework for Testing in Medicine) and deployed it on four large-language models to see how well they performed in settings closely mimicking actual interactions with patients.

All four large-language models did well on medical exam-style questions, but their performance worsened when engaged in conversations more closely mimicking real-world interactions.

This gap, the researchers said, underscores a two-fold need: First, to create more realistic evaluations that better gauge the fitness of clinical AI models for use in the real world and, second, to improve the ability of these tools to make diagnosis based on more realistic interactions before they are deployed in the clinic.

Evaluation tools like CRAFT-MD, the research team said, can not only assess AI models more accurately for real-world fitness but could also help optimize their performance in clinic.

“Our work reveals a striking paradox – while these AI models excel at medical board exams, they struggle with the basic back-and-forth of a doctor’s visit,” said study senior author Pranav Rajpurkar, assistant professor of biomedical informatics at Harvard Medical School. “The dynamic nature of medical conversations – the need to ask the right questions at the right time, to piece together scattered information, and to reason through symptoms – poses unique challenges that go far beyond answering multiple choice questions. When we switch from standardized tests to these natural conversations, even the most sophisticated AI models show significant drops in diagnostic accuracy.”

A better test to check AI’s real-world performance

Right now, developers test the performance of AI models by asking them to answer multiple choice medical questions, typically derived from the national exam for graduating medical students or from tests given to medical residents as part of their certification.

“This approach assumes that all relevant information is presented clearly and concisely, often with medical terminology or buzzwords that simplify the diagnostic process, but in the real world this process is far messier,” said study co-first author Shreya Johri, a doctoral student in the Rajpurkar Lab at Harvard Medical School. “We need a testing framework that reflects reality better and is, therefore, better at predicting how well a model would perform.”

CRAFT-MD was designed to be one such more realistic gauge.

To simulate real-world interactions, CRAFT-MD evaluates how well large-language models can collect information about symptoms, medications, and family history and then make a diagnosis. An AI agent is used to pose as a patient, answering questions in a conversational, natural style. Another AI agent grades the accuracy of final diagnosis rendered by the large-language model. Human experts then evaluate the outcomes of each encounter for ability to gather relevant patient information, diagnostic accuracy when presented with scattered information, and for adherence to prompts.

The researchers used CRAFT-MD to test four AI models — both proprietary or commercial and open-source ones — for performance in 2,000 clinical vignettes featuring conditions common in primary care and across 12 medical specialties.

All AI models showed limitations, particularly in their ability to conduct clinical conversations and reason based on information given by patients. That, in turn, compromised their ability to take medical histories and render appropriate diagnosis. For example, the models often struggled to ask the right questions to gather pertinent patient history, missed critical information during history taking, and had difficulty synthesizing scattered information. The accuracy of these models declined when they were presented with open-ended information rather than multiple choice answers. These models also performed worse when engaged in back-and-forth exchanges — as most real-world conversations are — rather than when engaged in summarized conversations.

Recommendations for optimizing AI’s real-world performance 

Based on these findings, the team offers a set of recommendations both for AI developers who design AI models and for regulators charged with evaluating and approving these tools.

These include:

• Use of conversational, open-ended questions that more accurately mirror unstructured doctor-patient interactions in the design, training, and testing of AI tools
• Assessing models for their ability to ask the right questions and to extract the most essential information
• Designing models capable of following multiple conversations and integrating information from them
• Designing AI models capable of integrating textual (notes from conversations) with and non-textual data (images, EKGs)
• Designing more sophisticated AI agents that can interpret non-verbal cues such as facial expressions, tone, and body language

Additionally, the evaluation should include both AI agents and human experts, the researchers recommend, because relying solely on human experts is labor-intensive and expensive.  For example, CRAFT-MD outpaced human evaluators, processing 10,000 conversations in 48 to 72 hours, plus 15-16 hours of expert evaluation. In contrast, human-based approaches would require extensive recruitment and an estimated 500 hours for patient simulations (nearly 3 minutes per conversation) and about 650 hours for expert evaluations (nearly 4 minutes per conversation). Using AI evaluators as first line has the added advantage of eliminating the risk of exposing real patients to unverified AI tools.

The researchers said they expect that CRAFT-MD itself will also be updated and optimized periodically to integrate improved patient-AI models.

“As a physician scientist, I am interested in AI models that can augment clinical practice effectively and ethically,” said study co-senior author Roxana Daneshjou, assistant professor of Biomedical Data Science and Dermatology at Stanford University.  “CRAFT-MD creates a framework that more closely mirrors real-world interactions and thus it helps move the field forward when it comes to testing AI model performance in health care.”

Authorship, funding, disclosures

Publication DOI 10.1038/s41591-024-03328-5

Additional authors included Jaehwan Jeong and Hong-Yu Zhou, Harvard Medical School; Benjamin A. Tran, Georgetown University; Daniel I. Schlessinger, Northwestern University; Shannon Wongvibulsin, University of California-Los Angeles; Leandra A. Barnes, Zhuo Ran Cai and David Kim, Sandford University; and Eliezer M. Van Allen, Dana-Farber Cancer Institute.

The work was supported by the HMS Dean’s Innovation Award and a Microsoft Accelerate Foundation Models Research grant awarded to Pranav Rajpurkar. SJ received further support through the IIE Quad Fellowship.

Daneshjou reported receiving personal fees from DWA, personal fees from Pfizer, personal fees from L’Oreal, personal fees from VisualDx, stock options from MDAlgorithms and Revea outside the submitted work, and a patent for TrueImage pending. Schlessinger is the co-founder of FixMySkin Healing Balms, a shareholder in Appiell Inc. and K-Health, a consultant with Appiell Inc and LuminDx, and an investigator for Abbvie and Sanofi. Van Allen serves as an advisor to Enara Bio, Manifold Bio, Monte Rosa, Novartis Institute for Biomedical Research, Serinus Bio. E.M.V.A provides research support to Novartis, BMS, Sanofi, NextPoint. Van Allen holds equity in Tango Therapeutics, Genome Medical, Genomic Life, Enara Bio, Manifold Bio, Microsoft, Monte Rosa, Riva Therapeutics, Serinus Bio, Syapse. Van Allen has filed for institutional patents on chromatin mutations and immunotherapy response, and methods for clinical interpretation; intermittent legal consulting on patents for Foaley & Hoag, and serves on the editorial board of Science Advances.

If ChatGPT were cut loose in the Emergency Department, it might suggest unneeded x-rays and antibiotics for some patients and admit others who didn’t require hospital treatment, a new study from UC San Francisco has found.

The researchers said that, while the model could be prompted in ways that make its responses more accurate, it’s still no match for the clinical judgment of a human doctor.

“This is a valuable message to clinicians not to blindly trust these models,” said postdoctoral scholar Chris Williams, MB BChir, lead author of the study, which appears Oct. 8 in Nature Communications. “ChatGPT can answer medical exam questions and help draft clinical notes, but it’s not currently designed for situations that call for multiple considerations, like the situations in an emergency department.”

Recently, Williams showed that ChatGPT, a large language model (LLM) that can be used for researching clinical applications of AI, was slightly better than humans at determining which of two emergency patients was most acutely unwell, a straightforward choice between patient A and patient B.

With the current study, Williams challenged the AI model to perform a more complex task: providing the recommendations a physician makes after initially examining a patient in the ED. This includes deciding whether to admit the patient, get x-rays or other scans, or prescribe antibiotics.

AI model is less accurate than a resident 

For each of the three decisions, the team compiled a set of 1,000 ED visits to analyze from an archive of more than 251,000 visits. The sets had the same ratio of “yes” to “no” responses for decisions on admission, radiology and antibiotics that are seen across UCSF Health’s Emergency Department.

Using UCSF’s secure generative AI platform, which has broad privacy protections, the researchers entered doctors’ notes on each patient’s symptoms and examination findings into ChatGPT-3.5 and ChatGPT-4. Then, they tested the accuracy of each set with a series of increasingly detailed prompts.

Overall, the AI models tended to recommend services more often than was needed. ChatGPT-4 was 8% less accurate than resident physicians, and ChatGPT-3.5 was 24% less accurate.

Williams said the AI’s tendency to overprescribe could be because the models are trained on the internet, where legitimate medical advice sites aren’t designed to answer emergency medical questions but rather to send readers to a doctor who can.

“These models are almost fine-tuned to say, ‘seek medical advice,’ which is quite right from a general public safety perspective,” he said. “But erring on the side of caution isn’t always appropriate in the ED setting, where unnecessary interventions could cause patients harm, strain resources and lead to higher costs for patients.”

He said models like ChatGPT will need better frameworks for evaluating clinical information before they are ready for the ED. The people who design those frameworks will need to strike a balance between making sure the AI doesn’t miss something serious, while keeping it from triggering unneeded exams and expenses.

This means researchers developing medical applications of AI, along with the wider clinical community and the public, need to consider where to draw those lines and how much to err on the side of caution.

“There’s no perfect solution,” he said, “But knowing that models like ChatGPT have these tendencies, we’re charged with thinking through how we want them to perform in clinical practice.”

Authors: Additional authors include Brenda Miao, Aaron Kornblith, and Atul Butte, all of UCSF.

Funding: The Eunice Kennedy Shriver National Institute of Child Health and Human Development and the National Institutes of Health (K23HD110716).

Disclosures: Please see the paper.

In a new study, researchers from the Mass General Brigham MESH Incubator found that ChatGPT, a generative artificial intelligence (AI) chatbot, showed promise as a tool to manage polypharmacy and deprescription. These findings, published April 18^th in the Journal of Medical Systems, demonstrate the first use case of AI models in medicine management.

To evaluate its utility, the investigators provided ChatGPT with different clinical scenarios and asked it a set of decision-making questions. Each scenario featured the same elderly patient taking a mixture of medications but included variations in cardiovascular disease history (CVD) and degree of impairment in activities of daily living (ADL).

When asked yes or no questions about reducing prescribed drugs, ChatGPT consistently recommended deprescribing medications in patients without a history of CVD. However, it was more cautious when overlying CVD was introduced, and more likely to keep the patient’s medication regimen unchanged. In both cases, the researchers observed that ADL impairment severity did not seem to affect decision outcomes.

The team also noted that ChatGPT had a tendency to disregard pain and favored deprescribing pain medications over other drug types like statins or antihypertensives. In addition, ChatGPT responses varied when presented with the same scenario in new chat sessions — which the authors suggest could reflect inconsistency in commonly reported clinical deprescribing trends on which the model was trained.

More than 40 percent of older adults meet the criteria for polypharmacy. The rate of seniors on Medicare seeing more specialists on their care teams has increased in recent years, leaving primary care providers to oversee medication management. An effective AI tool could help aid this practice, according to the researchers.

“Our study provides the first use case of ChatGPT as a clinical support tool for medication management,” said senior corresponding author Marc Succi, MD, Associate Chair of Innovation and Commercialization at Mass General Brigham Radiology and Executive Director of the MESH Incubator.  “While caution should be taken to increase accuracy of such models, AI-assisted polypharmacy management could help alleviate the increasing burden on general practitioners. Further research with specifically trained AI tools may significantly enhance the care of aging patients.”

Arya Rao, lead author, MESH researcher and Harvard Medical student, added “Our findings suggest that AI-based tools can play an important role in ensuring safe medication practices for older adults; it is imperative that we continue to refine these tools to account for the complexities of medical decision-making.”

Read more in the Journal of Medical Systems.

“For years, the debate has centred on how to improve the efficiency of healthcare. Thanks to advances in generative AI and language modelling, there are now opportunities to reduce the administrative burden on healthcare professionals. This will allow doctors to spend more time with their patients,” explains Cyrus Brodén, an orthopaedic physician and researcher at Uppsala University Hospital and Uppsala University.

Administrative tasks take up a large share of a doctor’s working hours, reducing the time for patient contact and contributing to a stressful work situation. Researchers at Uppsala University Hospital and Uppsala University, in collaboration with Danderyd Hospital and the University Hospital of Basel, Switzerland, have shown in a new study that the AI model ChatGPT can write administrative medical notes up to ten times faster than doctors without compromising quality.

The aim of the study was to assess the quality and effectiveness of the ChatGPT tool when producing medical record notes. The researchers used six virtual patient cases that mimicked real cases in both structure and content. Discharge documents for each case were generated by orthopaedic physicians. ChatGPT-4 was then asked to generate the same notes. The quality assessment was carried out by an expert panel of 15 people who were unaware of the source of the documents. As a secondary metric, the time required to create the documents was compared.

“The results show that ChatGPT-4 and human-generated notes are comparable in quality overall, but ChatGPT-4 produced discharge documents ten times faster than the doctors,” notes Brodén.

“Our interpretation is that advanced large language models like ChatGPT-4 have the potential to change the way we work with administrative tasks in healthcare. I believe that generative AI will have a major impact on healthcare and that this could be the beginning of a very exciting development,” he maintains.

The plan is to launch an in-depth study shortly, with researchers collecting 1,000 medical patient records. Again, the aim is to use ChatGPT to produce similar administrative notes in the patient records.

“This will be an interesting and resource-intensive project involving many partners. We are already working actively to fulfil all data management and confidentiality requirements to get the study under way,” concludes Brodén.

Rosenberg, G. S., Magnéli, M., Barle, N., Kontakis, M. G., Müller, A. M., Wittauer, M., Gordon, M., & Brodén, C.. 2024: ChatGPT-4 generates orthopedic discharge documents faster than humans maintaining comparable quality: a pilot study of 6 cases. Acta Orthopaedica, 95, 152-156. https://doi.org/10.2340/17453674.2024.40182

An interdisciplinary study led by University of California, Riverside, computer scientists found that both internet information gathering services have strengths and weaknesses for people seeking information about Alzheimer’s disease and other forms of dementia. The team included clinical scientists from the University of Alabama and Florida International University.

Google provides the most current information, but query results are skewed by service and product providers seeking customers, the researchers found. ChatGPT, meanwhile, provides more objective information, but it can be outdated and lacks the sources of its information in its narrative responses.

“If you pick the best features of both, you can build a better system, and I think that this is what will happen in the next couple of years,” said Vagelis Hristidis, a professor of computer science and engineering in UCR’s Bourns College of Engineering.

In their study, Hristidis and his co-authors submitted 60 queries to both Google and ChatGPT that would be typical submissions from people living with dementia and their families.

The researchers focused on dementia because more than 6 million Americans are impacted by Alzheimer’s disease or a related condition, said study co-author Nicole Ruggiano, a professor of social work at the University of Alabama.

“Research also shows that caregivers of people living with dementia are among the most engaged stakeholders in pursuing health information, since they often are tasked with making decisions for their loved one’s care,” Ruggiano said.

Half of the queries submitted by the researchers sought information about the disease processes, while the other half sought information on services that could assist patients and their families.

The results were mixed.

“Google has more up-to-date information, and covers everything,” Hristidis said. “Whereas ChatGPT is trained every few months. So, it is behind. Let’s say there’s some new medicine that just came out last week, you will not find it on ChatGPT.”

While dated, ChatGPT provided more reliable and accurate information than Google. This is because the ChatGPT creators at OpenAI choose the most reliable websites when they train ChatGPT through computationally intensive machine learning. Yet, users are left in dark about specific sources of information because the resulting narratives are void of references.

Google, however, has a reliability problem because it essentially “covers everything from the reliable sources to advertisements,” Hristidis said.

In fact, advertisers pay Google for their website links to appear at the top of search result pages. So, users often first see links to websites of for-profit companies trying to sell them care-related services and products. Finding reliable information from Google searches thus requires a level of user skill and experience, Hristidis said.

Co-author Ellen Brown, an associate professor of nursing at the Florida International University, pointed out that families need timely information about Alzheimer’s.

“Although there is no cure for the disease, many clinical trials are underway and recently a promising treatment for early stage Alzheimer’s disease was approved by the FDA,” Brown said. “Therefore, up-to-date information is important for families looking to learn about recent discoveries and available treatments.”

The authors of the study write that “the addition of both the source and the date of health-related information and availability in other languages may increase the value of these platforms for both non-medical and medical professionals.” It was published in the Journal of Medical Internet Research under the title “ChatGPT vs Google for Queries Related to Dementia and Other Cognitive Decline: Comparison of Results.”

Google and ChatGPT both scored low for readability scores, which makes it difficult for people with lower levels of education and low health literacy skills.

“My prediction is that the readability is the easier thing to improve because there are already some tools, some AI methods, that can read and paraphrase text,” Hristidis said. “In terms of improving reliability, accuracy, and so on, that’s much harder. Don’t forget that it took scientists many decades of AI research to build ChatGPT. It is going to be slow improvements from where we are now.”