Fortunately, we are not quite there yet, but digitalization is advancing rapidly in the healthcare sector as well. AI chatbots and digital symptom checkers are playing an increasingly important role and are more and more serving as the first point of contact for so-called “self-triage”—that is, the initial assessment of the urgency of treatment by the patients themselves.

But while the technical capabilities of these systems are constantly growing, another factor is coming into the focus of research: how humans communicate with the machine. This is an important topic because even the best technology, especially in medical diagnostics, relies on precise information that users do not always provide in full.

Human reluctance limits the potential of AI

This is the central finding of a study now published in the journal Nature Health. The study was led by Professor Wilfried Kunde, holder of the Chair of Psychology III at the University of Würzburg, and Moritz Reis, a research associate in that department. It involved scientists from Charité – Universitätsmedizin Berlin, the University of Cambridge, as well as Helios Klinikum Emil von Behring and Vivantes Klinikum Neukölln in Berlin.

“The 500 study participants were tasked with writing simulated symptom reports for two common conditions – unusual headaches and flu-like symptoms” describes lead author Moritz Reis the study design. They were led to believe that their reports would be read either by an AI chatbot or a human doctor. The goal was to examine the quality of these reports in terms of their suitability for a medical urgency assessment.

Loss of quality is evident in reduced level of detail

The key finding: When participants believed they were communicating with artificial intelligence, the suitability of their descriptions for an initial medical assessment deteriorated measurably compared to interactions with supposed medical professionals. This effect was even observed among participants who were actually experiencing the relevant symptoms at the time of the survey.

This loss of quality is directly reflected in the level of detail in the reports. While descriptions provided to medical professionals averaged 255.6 characters, those provided to chatbots averaged only 228.7 characters.

Even though a difference of 28 characters may sound small, the research team states that this effect is practically relevant and can result in even high-performance AI models ultimately providing incorrect medical advice. After all, these models also fail to make an accurate medical assessment if patients do not provide all essential information. The success of digital initial assessments depends less on computational power than on the patient’s willingness to provide a detailed description.

Psychological Barriers: Concerns About a “One-Size-Fits-All Diagnosis”

But why are people so hesitant when it comes to machines? A key reason is likely what’s known as “uniqueness neglect.” “Many people assume that AI cannot grasp the individual nuances of their personal situation and instead merely matches standardized patterns,” explains Wilfried Kunde.

In addition, skepticism about algorithms’ diagnostic capabilities, as well as privacy concerns, may lead people to provide abbreviated or vague information. Moritz Reis sums up the human component this way: “If we don’t trust a machine to understand our uniqueness, we may unconsciously withhold the information it would need to provide precise assistance.” This psychological filter can have the effect that medically relevant details never even reach the system, thereby lowering the quality of the diagnosis.

Improving the dialogue with the machine

In the research team’s view, the findings clearly show that the technical advancement of AI alone is not sufficient. They therefore see a potential solution in the intelligent design of user interfaces.

To improve the quality of symptom reports, developers should provide concrete examples of high-quality descriptions and program the AI to actively request missing details. Only when users are encouraged to provide detailed information misdiagnoses can be avoided and the burden on the healthcare system could be effectively reduced.

In an expert commentary titled ‘AI can reason like a physician; what comes next? published in Science, Flinders researchers caution that while new AI systems show impressive capabilities, strong results in controlled studies do not mean they are ready for routine use in hospitals or clinics.

The authors say there is an urgent need to understand how emerging AI tools can be safely integrated into everyday clinical practice, with patient outcomes remaining the central focus.

Despite these warnings, the researchers acknowledge that recent advances in AI create genuine opportunities to support doctors, particularly in busy and high-pressure care settings.

The commentary reviews new research showing that advanced reasoning-based AI systems can work through diagnostic scenarios step by step and, in some cases, closely match or even exceed the diagnostic performance of experienced doctors.

Erik Cornelisse, a PhD candidate at Flinders University and co-author of the commentary, says this shift marks a move from simple question answering tools towards algorithms capable of seemingly human-like clinical reasoning on text-based tasks.

However, the Flinders team stresses that real world medical care involves far more than text-based reasoning or test performance.

They say clinical practice depends on physical examination, listening to patients, understanding medical and social context, and taking responsibility for outcomes, elements that current AI systems cannot safely provide on their own.

“Health care decisions are complex, high stakes, and deeply human, and accuracy alone, particularly on just text-based cases, does not make a system safe for patients,” says Mr Cornelisse from the College of Medicine and
Public Health.

Senior author Associate Professor Ash Hopkins, an NHMRC Investigator and leader of Flinders’ Clinical Cancer Epidemiology Lab, says modern healthcare relies on judgement, accountability, and ethical oversight.

“AI systems have demonstrated that they can reason through clinical problems with similar performance to doctors, notably on the same scenarios used to train clinicians themselves. This presents genuine opportunities to support clinicians in the future,” says Associate Professor Hopkins.

“Multiple stakeholders are currently working on the frameworks for AI in terms of legal, professional, or moral responsibility for its decisions, and presently there is a critical need for deliberate and controlled integration into clinical care.”

The commentary highlights known risks linked to poorly evaluated systems, including bias, inequitable care, and unintended patient harm.

“History shows that algorithms can worsen outcomes when deployed without sufficient safeguards and can amplify problems as easily as they solve them, particularly when systems are trained on incomplete or unrepresentative data,” says Mr Cornelisse.

Looking ahead, the Flinders researchers argue that enthusiasm for medical AI must be matched by strong governance and clearer standards for evaluation.

“We do not allow doctors to practise without supervision and evaluation, and AI should be held to comparable standards,” says Mr Cornelisse.

The researchers stress that improvement in real patient outcomes, not exam scores, benchmarks, or demonstrations, must be the true measure of success.

Associate Professor Hopkins says AI holds enormous promise but must be applied responsibly.

“Patients deserve technology that improves care in the real world, not systems that only look impressive in studies,” he says.

“With careful design, strong oversight, and rigorous evaluation, AI could become a powerful tool to deliver safer, fairer, and more effective care across health systems worldwide,” concludes Associate Professor Hopkins.

The paper, ‘AI can reason like a physician; what comes next?’, by Ashley M. Hopkins and Erik Cornelisse is published in SCIENCE.  DOI 10.1126/science.aeg8766 (link live after embargo lifts)

The findings are based on a nationally representative survey of more than 5,500 U.S. adults conducted from October through December 2025.

In the past 30 days, did you use an AI tool or chatbot for health-related information or advice for any of the following reasons?

% Yes, among adults who have used AI tools or chatbots for health-related information or advice in the past 30 days

Note. Categories are for descriptive purposes only and were not shown on the survey.

Among Americans who have used AI for health-related information or advice in the past 30 days, the most frequently cited motivations are wanting answers quickly (71%) and wanting additional information (71%). Nearly seven in 10 (67%) say they were curious about what AI would say, and roughly six in 10 report using AI to do research on their own before (59%) or after (56%) seeing a doctor.

Regardless of the reason, almost half (46%) of Americans who used AI for healthcare information say the AI tool or chatbot made them feel more confident talking with or asking questions of a provider. Others say it helped them identify issues earlier (22%) or avoid unnecessary medical tests or procedures (19%).

“Artificial intelligence is already reshaping how Americans seek health information, make decisions and engage with providers, and health systems must keep pace,” said Tim Lash, President, West Health Policy Center, a nonprofit and nonpartisan organization focused on aging and healthcare affordability. “The risk isn’t that AI is moving too fast — it’s that health systems may move too slowly to guide its use in healthcare responsibly.”

A Smaller Share Turn to AI in Place of a Provider

While self-directed research is the primary driver of AI health use, a smaller but notable share of recent users report turning to AI instead of seeing a healthcare provider, particularly when faced with cost, access or quality barriers. Among recent AI health users, 27% say they didn’t want to pay for a doctor’s visit and 14% say they were unable to pay. One in five (21%) say they didn’t have time to make an appointment, and 16% say they couldn’t access a doctor or provider. Another 21% say they felt dismissed or ignored by a provider in the past, and 18% say they were too embarrassed to talk to a person.

In the past 30 days, did you use an AI tool or chatbot for health-related information or advice for any of the following reasons?

% Yes, among adults who have used AI for health-related information and advice in the past 30 days

Among recent AI health users, 84% still saw a healthcare provider, but 14% report not seeing a provider they otherwise would have seen because of information or advice they received from AI. When projected to the full U.S. adult population, this represents roughly 14 million Americans who did not see a provider after receiving AI-generated health information.

Trust in that AI-generated health information, however, remains divided. Among those who consulted it in the past 30 days, roughly one-third say they trust it (33%), one-third neither trust nor distrust it (33%), and about one-third distrust it (34%). However, only 4% say they strongly trust the accuracy, indicating that many Americans are making healthcare decisions based on AI-generated information without full confidence in its accuracy.

About one in 10 (11%) who report using AI for health information or advice in the past 30 days say that AI recommended healthcare information or advice they believed was unsafe.

“This data indicates that while some Americans may be using artificial intelligence as a substitute for going to the doctor’s office, many see it as a tool to complement their healthcare, helping them understand symptoms they might be feeling and clarify any diagnosis they receive from their doctors,” said Joe Daly, Global Managing Partner at Gallup.

Motivations Vary by Age and Income

While information-seeking is the dominant reason Americans turn to AI for health purposes, use patterns differ by demographics. Younger adults are more likely than older adults to use AI for self-directed research — 69% of adults aged 18 to 29 say they do research before seeing a doctor, compared with 43% of those 65 and older.

Income differences are most visible in barrier-driven motivations. Among adults earning less than $24,000 annually, 32% say they used AI because they could not pay for a doctor’s visit, compared with just 2% among those earning $180,000 or more.

Everyday Health Questions Top the List of AI Use Cases

Americans who used AI for health information or advice in the past 30 days most often report using it to gather information about everyday health concerns, including physical symptoms (58%) and nutrition or exercise (59%). But AI use extends beyond symptom-checking — Americans who used AI in the past 30 days also report using AI to understand medication side effects (46%), interpret medical information (44%), or research a diagnosis or medical condition (38%). Nearly one in four (24%) report using AI to explore mental health or emotional concerns.

Methodology

West Health-Gallup Center on Healthcare, October-December 2025

Results are based on a Gallup Panel study conducted Oct. 27-Dec. 22, 2025, with a sample of 5,660 adults aged 18 and older who are members of the Gallup Panel, a nationally representative, probability-based panel of U.S. adults. Gallup uses random selection methods to recruit Panel members, including random-digit-dial (RDD) phone interviews that cover landlines and cellphones and address-based sampling (ABS) methods. Respondents with internet access completed the questionnaire as a web survey, and those without regular internet access were sent a printed questionnaire to complete and return by mail. The sample for this study was weighted to be demographically representative of the U.S. adult population, using the most recent Current Population Survey figures. For results based on this sample, one can say that the maximum margin of sampling error is ±2.1 percentage points at the 95% confidence level. Margins of error are higher for subsamples. In addition to sampling error, question wording and practical difficulties in conducting surveys can introduce error and bias into the findings of public opinion polls.

The authors show that existing regulatory requirements for approval have so far only partially addressed many of these so-called “human factors-related risks”. This can create gaps that impact the safety and quality of care. To address these, the researchers identify seven key risks and develop practical recommendations for action that can be integrated into existing regulatory and documentation processes.

Risks in the use of AI systems

AI-based medical devices can be used in various areas of clinical environments. In radiology, for example, they assist in detecting cancer. Clinical decision support systems help select personalized therapies for patients.  AI can also support real-time monitoring and early warning systems, as well as chatbots for applications such as patient communication and software that automatically generate medical reports or summarize findings. The analysis focuses on risks that may arise in the practical use of such AI systems. These include, for example, an increased likelihood of outputs being misunderstood or misinterpreted due to the sometimes-opaque nature of AI systems. Problems can also occur when trust in the application is miscalibrated: resulting in users either relying too heavily on AI assistance or ignoring relevant recommendations. The researchers also point to the risk of automation bias: the tendency to uncritically adopt recommendations from automated systems, potentially overlooking errors or forgoing independent judgment. Additional risks include potential deskilling, technostress among users, an unchecked expansion of indications beyond the originally intended scope (indication creep), and errors related to system changes or different operating modes. Such factors can create additional burdens or unexpected failures in clinical practice – even when the technical performance of a system itself is strong.

A practical guide for manufacturers and evaluators

For their analysis, the research team evaluated existing standards on usability and safety, regulatory guidelines, alongside the scientific literature on AI in healthcare. In addition, expert discussions from the fields of clinical application, regulation, and human factors were incorporated. The result is a practical guide, that fills a gap in current standards, with seven recommendations. These are intended to support manufacturers and evaluators both before and after a product is placed on the market. The aim is to identify AI-specific risks in interaction with human users at an early stage and to address them systematically.

The framework recommends developing and deploying AI-based medical devices in a way that clearly defines the users, in which context the systems are applied, and which tasks are assigned to humans and which to the system. Furthermore, results should be presented in a way that is easy to understand, integrated into existing clinical workflows, and supplemented by training where needed as well as safe fallback options in the event of system failures. The authors emphasize the importance of continuous monitoring after market entry. Usage patterns, potential misuse, or overreliance on AI systems should be systematically observed and corrected as needed. Changes to the systems must also be communicated transparently so that work processes can be adjusted accordingly.

The recommendations are deliberately formulated in general but regulatory-aligned terms so that they can be applied to different AI-enabled medical devices and application scenarios. In a next step, the researchers aim to test and further develop their recommendations based on concrete pilot applications with AI-enabled medical devices. In the long term, human factors should be systematically considered in the regulation and evaluation of AI-based health technologies – reducing avoidable risks while supporting safe innovation in medicine.

The article was authored by researchers from TU Dresden (EKFZ for Digital Health, Chair of Industrial Design Engineering, and Faculty of Business and Economics), in collaboration with experts from the University of Oxford (United Kingdom) and Geneva University Hospital (Switzerland).

Publication

Rebecca Mathias, Anne Schmitt, Mateo Campos, Baptiste Vasey, Sebastian Lorenz, Peter McCulloch, Stephen Gilbert: Evaluation of Human Factors-Related Risks in AI-Enabled Medical Devices: A Practical Guide, NEJM AI, 2026. Link: https://ai.nejm.org/doi/full/10.1056/AIpc2501297

Else Kröner Fresenius Center (EKFZ) for Digital Health

The EKFZ for Digital Health at the Faculty of Medicine at TUD Dresden University of Technology and University Hospital Carl Gustav Carus Dresden was established in September 2019. It receives funding of around 40 million euros from the Else Kröner Fresenius Foundation for a period of ten years. The center focuses its research activities on innovative, medical and digital technologies at the direct interface with patients. The aim here is to fully exploit the potential of digitalization in medicine to significantly and sustainably improve healthcare, medical research and clinical practice.

A new study, published in the March 9 online issue of npj Health Systems [https://doi.org/10.1038/s44401-026-00077-0], suggests that the answer depends less on the AI itself and more on how it’s designed.

The investigators found that health care AI systems work far better when tasks are distributed among multiple specialized AI “agents”—software systems that can perform complex tasks, learn, and adapt—rather than relying on a single, all-purpose agent. This multi-agent approach kept performance steady even as demands increased, while dramatically reducing computing costs and delays, say the investigators.

“For health care organizations, our findings point to a smarter way to use AI,” says senior study author Girish N. Nadkarni, MD, MPH, Barbara T. Murphy Chair of the Windreich Department of Artificial Intelligence and Human Health, Director of the Hasso Plattner Institute for Digital Health, Irene and Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine, and Chief AI Officer of the Mount Sinai Health System. “By assigning different tasks, such as finding patient information, extracting data, or checking medication doses, to specialized AI agents, systems can run faster and more reliably while keeping costs under control. Ultimately, this kind of design could help health care teams spend less time on administrative work and more time focusing on patients.”

As part of the study, the researchers compared two approaches to clinical AI: a single system responsible for handling many different clinical tasks, and a coordinated network of specialized AI agents overseen by a central “orchestrator.” Using state-of-the-art language models, the team evaluated performance across common clinical functions, including information retrieval, data extraction, and medication dosing calculations—under simulated real-world conditions involving up to 80 simultaneous tasks.

“What we found is that AI systems behave a lot like people,” says study lead author Eyal Klang, MD, formerly with the Icahn School of Medicine. “When you ask one system to do too many different things at once, performance suffers. But when one orchestrator agent divides the work among specialized agents, the system stays accurate, responsive, and far more efficient, even under heavy demand.”

The coordinated multi-agent system maintained superior accuracy levels while using far fewer computing resources, up to 65 times fewer, than a single-agent design. The study simulated real clinical “traffic,” where many types of tasks arrive at once and compete for attention, the investigators say.

“Our findings show that smart coordination is not just a technical preference,” Dr. Klang says. “It can make the difference between an AI system that continues to function smoothly and one that begins to break down when it is exposed to the pressures of real clinical workloads.”

Next, the research team plans to test these coordinated AI systems directly in clinical settings, using real-time patient data. If successful, this approach could help shape how hospitals and health systems scale AI in the future, helping them handle peak workloads without sacrificing quality or safety.

The researchers emphasize that the gains are not automatic: even sophisticated AI can fall short when systems are poorly designed or implemented. “Health care does not operate one task at a time,” Dr. Nadkarni says. “Hospitals face constant, overlapping demands, especially during busy periods. Our findings show that the future of health care AI is not a single super-intelligent system, but a coordinated team of focused agents that work together to scale safely, control costs, and support real clinical operations.”

“When a single agent handles everything, you can’t trace where it went wrong. With the orchestrator, every step is logged, which tool was called, what it returned, and how the answer was assembled. At 80 simultaneous tasks, the single agent dropped to 16 percent accuracy while burning 65 times more compute—and you’d have no way to figure out why. That kind of transparency isn’t optional in medicine,” says second author Mahmud Omar, MD, a visiting researcher in the Windreich Department. “This matters more now than ever—agentic AI is no longer a research concept. Tools like OpenAI’s operator mode, Claude’s Cowork, and similar platforms are putting autonomous agents directly in the hands of clinicians and patients. As that adoption accelerates, the architecture behind these systems has to be auditable from the start.”

The paper is titled “Orchestrated multi agents sustain accuracy under clinical‑scale workloads compared to a single agent.”

The study’s authors, as listed in the journal, are Eyal Klang, Mahmud Omar, Ganesh Raut, Reem Agbareia, Prem Timsina, Robert Freeman, Lisa Stump, Alexander Charney, Benjamin S. Glicksberg, Girish N. Nadkarni.

This work was supported  in part by the Clinical and Translational Science Awards (CTSA) grant UL1TR004419 from the National Center for Advancing Translational Sciences. Research reported in this publication was also supported by the Office of Research Infrastructure of the National Institutes of Health under award number S10OD026880 and S10OD030463.

For more Mount Sinai artificial intelligence news, visit: https://icahn.mssm.edu/about/artificial-intelligence.

Using a large language model (LLM), the MIT team analyzed the genetic code of the industrial yeast Komagataella phaffii — specifically, the codons that it uses. There are multiple possible codons, or three-letter DNA sequences, that can be used to encode a particular amino acid, and the patterns of codon usage are different for every organism.

The new MIT model learned those patterns for K. phaffii and then used them to predict which codons would work best for manufacturing a given protein. This allowed the researchers to boost the efficiency of the yeast’s production of six different proteins, including human growth hormone and a monoclonal antibody used to treat cancer.

“Having predictive tools that consistently work well is really important to help shorten the time from having an idea to getting it into production. Taking away uncertainty ultimately saves time and money,” says J. Christopher Love, the Raymond A. and Helen E. St. Laurent Professor of Chemical Engineering at MIT, a member of the Koch Institute for Integrative Cancer Research, and faculty co-director of the MIT Initiative for New Manufacturing (MIT INM).

Love is the senior author of the new study, which appears this week in the Proceedings of the National Academy of Sciences. Former MIT postdoc Harini Narayanan is the paper’s lead author.

Codon optimization

Yeast such as K. phaffii and Saccharomyces cerevisiae (baker’s yeast) are the workhorses of the biopharmaceutical industry, producing billions of dollars of protein drugs and vaccines every year.

To engineer yeast for industrial protein production, researchers take a gene from another organism, such as the insulin gene, and modify it so that the microbe will produce it in large quantities. This requires coming up with an optimal DNA sequence for the yeast cells, integrating it into the yeast’s genome, devising favorable growth conditions for the it, and finally purifying the end product.

For new biologic drugs — large, complex drugs produced by living organisms — this development process might account for 15 to 20 percent of the overall cost of commercializing the drug.

“Today, those steps are all done by very laborious experimental tasks,” Love says. “We have been looking at the question of where could we take some of the concepts that are emerging in machine learning and apply them to make different aspects of the process more reliable and simpler to predict.”

In this study, the researchers wanted to try to optimize the sequence of DNA codons that make up the gene for a protein of interest. There are 20 naturally occurring amino acids, but 64 possible codon sequences, so most of these amino acids can be encoded by more than one codon. Each codon corresponds to a unique transfer RNA (tRNA) molecule, which carries the correct amino acid to the ribosome, where amino acids are strung together into proteins.

Different organisms use each of these codons at different rates, and designers of engineered proteins often optimize the production of their proteins by choosing the codons that occur the most frequently in the host organism. However, this doesn’t necessarily produce the best results. If the same codon is always used to encode arginine, for example, the cell may run low on the tRNA molecules that correspond to that codon.

To take a more nuanced approach, the MIT team deployed a type of large language model known as an encoder-decoder. Instead of analyzing text, the researchers used it to analyze DNA sequences and learn the relationships between codons that are used in specific genes.

Their training data, which came from a publicly available dataset from the National Center for Biotechnology Information, consisted of the amino acid sequences and corresponding DNA sequences for all of the approximately 5,000 proteins naturally produced by K. phaffii.

“The model learns the syntax or the language of how these codons are used,” Love says. “It takes into account how codons are placed next to each other, and also the long-distance relationships between them.”

Once the model was trained, the researchers asked it to optimize the codon sequences of six different proteins, including human growth hormone, human serum albumin, and trastuzumab, a monoclonal antibody used to treat cancer.

They also generated optimized sequences of these proteins using four commercially available codon optimization tools. The researchers inserted each of these sequences into K. phaffii cells and measured how much of the target protein each sequence generated. For five of the six proteins, the sequences from the new MIT model worked the best, and for the sixth, it was the second-best.

“We made sure to cover a variety of different philosophies of doing codon optimization and benchmarked them against our approach,” Narayanan says. “We’ve experimentally compared these approaches and showed that our approach outperforms the others.”

Learning the language of proteins

K. phaffii, formerly known as Pichia pastoris, is used to produce dozens of commercial products, including insulin, hepatitis B vaccines, and a monoclonal antibody used to treat chronic migraines. It is also used in the production of nutrients added to foods, such as hemoglobin.

Researchers in Love’s lab have started using the new model to optimize proteins of interest for K. phaffii, and they have made the code available for other researchers who wish to use it for K. phaffii or other organisms.

The researchers also tested this approach on datasets from different organisms, including humans and cows. Each of the resulting models generated different predictions, suggesting that species-specific models are needed to optimize codons of target proteins.

By looking into the inner workings of the model, the researchers found that it appeared to learn some of the biological principles of how the genome works, including things that the researchers did not teach it. For example, it learned not to include negative repeat elements — DNA sequences that can inhibit the expression of nearby genes. The model also learned to categorize amino acids based on traits such as hydrophobicity and hydrophilicity.

“Not only was it learning this language, but it was also contextualizing it through aspects of biophysical and biochemical features, which gives us additional confidence that it is learning something that’s actually meaningful and not simply an optimization of the task that we gave it,” Love says.

###

The research was funded by the Daniel I.C. Wang Faculty Research Innovation Fund at MIT, the MIT AltHost Research Consortium, the Mazumdar-Shaw International Oncology Fellowship, and the Koch Institute.

The global scoping review looked at this early phase of testing and revealed huge variations in the way the trials are being conducted and the measures used to assess the effectiveness of the tools.

“This lack of guidance around silent trials is concerning as AI models can be unpredictable and difficult to use in real-world settings if they haven’t been tested thoroughly,” said corresponding author Lana Tikhomirov, a PhD candidate from Adelaide University’s Australian Institute for Machine Learning.

“Some of the trials in our review focused on AI metrics that weren’t clinically useful, while others looked at the bare minimum with no details on how the model performed in a clinical setting.

“If these AI tools are rolled out without comprehensive testing and things go wrong, it could expose both patients and clinicians to harmful advice.”

Silent trials are when AI models are tested in their intended setting for use, but the results don’t influence patient care as they aren’t given to the clinical team at the time of treatment.

Currently there are no formal guidelines on how to conduct these trials, which researchers say are critical to ensure an AI tool will be useful and beneficial in a local setting.

“Silent trials are a low-risk way to test technology without compromising patient outcomes,” said co-author Associate Professor Melissa McCradden, who is the Deputy Director of Adelaide University’s Australian Institute for Machine Learning, AI Director at the Women’s and Children’s Health Network and Hospital Research Foundation Fellow in Paediatric AI Ethics.

“We know that many AI models fail when they’re introduced into real-world settings and an AI tool that works in one hospital may not work in another.

“Conducting comprehensive silent trials that adhere to a clear set of international guidelines is critical if we want to successfully take AI tools from bench to bedside.”

The scoping review has been published in Nature Health and is part of a larger study looking at silent phase evaluations for healthcare AI.

Project CANAIRI – Collaboration for Translational Artificial Intelligence Trials – is focusing on developing guidance for silent trials to ensure health settings in which AI is intended to be used are ready and able to do so in a beneficial way.

“Ultimately, we would like to see silent trials become a mandatory part of the process of adopting AI tools in medicine,” said Associate Professor McCradden, who is the lead of Project CANAIRI.

An interdisciplinary George Mason University team led by Farrokh Alemi, an expert in machine-learning and AI, compared how effective recommendations were from AI-guided tools/models that knew the patient’s race and factors uniquely relevant to African American patients against tools/models that didn’t. The team found that recommendations based on “race-blind” AI models—those that do not know the patient’s race—tended to recommend medications that were less effective for African American patients.

“Anti-depressant recommendations from race-specific models outperformed recommendations from general models across all antidepressants studied. The findings highlight why clinical AI, like clinical practice, shouldn’t rely solely on general-population patterns when prescribing for African Americans with depression,” said Vladimir Cardenas, master of science in health informatics ’24.

Why This Matters

“If AI systems are not trained on correct information, including patient demographic information, such as race, it will give incorrect or inaccurate information, which can result in people ending up with less effective medications,” said Alemi.

Alemi and his co-researchers observed that when advising patients on options for treating depression. “AI systems could be biased against African Americans, recommending antidepressants that work for general, mostly White, patients but not for African Americans,” said Alemi.

The Details

Researchers looked at bias in an AI system meant to guide treatment for Major Depressive Disorder (MDD)—and whether race-blind models miss important signals for African American patients. The AI system used medical history—including whether a patient completed the full dose of the antidepressant—to recommend a medication. Researchers coded whether a patient discontinued the use of the antidepressant as a measure of AI-guided treatment failure or success.

The study underscores that race is not a biological determinant of depression or treatment response, emphasizing the social and environmental factors that affect depression. Some of these factors more common among African American patients may be poverty, low education, exposure to violence, discrimination, cultural stigma and negative attitudes toward mental health, and low access to mental health treatment resources.

“These data highlight the need to tailor antidepressants to fit the patient’s individual medical history. Clinicians do this, and, if done right, an AI system can help clinicians do so as well,” said Cardenas.

“I hope that our approach will help inform AI in health care design and governance. This way we can truly pursue AI that improves the health of all,” said Cardenas.

The research team included Kevin Lybarger, assistant professor in the College of Engineering and Computing, along with master of science in health informatics graduates Cardenas, Maria Kurian, and Rachel Christine King; and Niloofar Ramezani from Virginia Commonwealth University.

Bias in AI-guided management of patients with major depressive disorders was published in the Journal of Health Equity in January 2026. The study was supported by the Artificial Intelligence/Machine Learning Consortium to Advance Health Equity and Researcher Diversity. Research was partially funded through a Patient-Centered Outcomes Research Institute (PCORI) Award.

Key Takeaways

• AI systems for antidepressant guidance may be less effective for African American patients because models use data from general, primarily White, populations.
• Race-specific models were more accurate in predicting African Americans’ responses to medications across all antidepressants studied.
• Clinical AI treating mental health shouldn’t rely solely on general population data when prescribing antidepressants for African Americans with depression.

Amid growing adoption of new AI tools across the industry, the regulators have set out broad guidance on how the technology can be used to generate evidence, manufacture products and monitor safety.

The principles are relevant for those developing medicines, as well as for marketing authorisation applicants and holders, the regulators said in a joint statement. They will underpin future AI guidance in both jurisdictions and support enhanced international collaboration. The development of formal guidelines in the EU is already underway, building on the EMA AI reflection paper published in 2024.

‘The principles are a good showcase of how we can work together on the two sides of the Atlantic to preserve our reading role in the global innovation race, while ensuring the highest level of patient safety,’ said European Commission for Health and Animal Welfare, Olivér Várhelyi.

The European Commission’s Biotech Act proposal, published in December, notes the promise of AI as a tool to accelerate innovation. New pharmaceutical legislation, meanwhile, aims to accommodate the broader use of AI in regulatory decision-making, and creates additional possibilities for testing AI-driven methods for medicines in a controlled environment.

To realise these benefits, say regulators, AI needs to be expertly managed, including the mitigation of risks. Ethics should be at the forefront of policymaking and regulations, it noted.

As AI continues to evolve, a principles-based approach will help regulators, pharmaceutical companies and medicines developers harness the potential of these technologies while ensuring patient safety and regulatory compliance.

To diagnose a mental illness, medical practitioners rely on a variety of factors, including the symptoms reported by patients and recorded on clinical questionnaires. The precise wording of individual questions on these questionnaires is often crucial for making the correct diagnosis. However, standard questionnaires often vary considerably. Researchers have found evidence of overlaps and deviations in the content of questions used to identify depression, bipolar disorder, and the risk of psychosis, which makes accurate diagnosis difficult.

In addition, doctors rely on their clinical experience. This means that they associate individual symptoms with a specific illness that corresponds to their experience. However, as different illnesses can produce the same or similar symptoms, this can also increase the risk of misdiagnosis. “We know surprisingly little about whether – and how – the wording of clinical questionnaires triggers certain associations in doctors,” says Professor Joseph Kambeitz. Inconsistent findings could also result from differences among patients in the same diagnostic group or, alternatively, from differences between questionnaires.

Using large language models (LLMs) is one approach to analysing language-mediated illness descriptions. The team used the LLMs GPT-3, Llama and BERT to analyse both the structure and content of four clinical questionnaires. The study was based on data from over 50,000 questionnaires on depression, anxiety, psychosis risk, and autism.

In clinical practice, symptoms often occur simultaneously, such as the empirical association between a lack of drive and a loss of pleasure. The analysis showed that the LLMs ‘recognize’ which symptoms commonly occur together. Even without access to specific empirical data, the same symptom associations are evident in LLMs based purely on the questionnaire formulations.

This suggests new ways in which artificial intelligence could improve psychological questionnaires in future, by avoiding redundant items and making diagnosis and understanding of mental illnesses more efficient. LLMs can be used to develop questionnaires that are both precise (i.e. that reliably recognize psychological symptoms) and efficient, asking only as many questions as necessary in order to simplify the process for patients and practitioners.

“AI can map both medical knowledge and the structures of mental illnesses. This is an important step in bringing digital methods and neuroscience closer together, and in advancing the development of diagnostics and research in psychiatry,” says Professor Kai Vogeley.

Professor Joseph Kambeitz concludes: “In psychiatry, the ‘spoken word’ plays an important role in diagnosis and therapy. There are currently many promising projects that are investigating how we can use LLMs in psychiatry, from diagnostics via the writing and amending of reports to the simulation of therapy sessions. We can expect many more exciting research results in this field.”

Now, researchers at the Icahn School of Medicine at Mount Sinai and collaborators have developed an artificial intelligence system, called InfEHR, that links unconnected medical events over time, creating a diagnostic web that reveals hidden patterns. Published in the September 26 online issue of Nature Communications, the study shows that Inference on Electronic Health Records (InfEHR) transforms millions of scattered data points into actionable, patient-specific diagnostic insights.

“We were intrigued by how often the system rediscovered patterns that clinicians suspected but couldn’t act on because the evidence wasn’t fully established,” says senior corresponding author Girish N. Nadkarni, MD, MPH, Chair of the Windreich Department of Artificial Intelligence and Human Health, Director of the Hasso Plattner Institute for Digital Health, the Irene and Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai, and the Chief AI Officer of the Mount Sinai Health System. “By quantifying those intuitions, InfEHR gives us a way to validate what was previously just a hunch and opens the door to entirely new discoveries.”

Most medical artificial intelligence (AI), no matter how advanced, applies the same diagnostic process to every patient. InfEHR works differently by tailoring its analysis to each individual. The system builds a network from a patient’s specific medical events and their connections over time, allowing it to not only provide personalized answers but also to ask personalized questions. By adapting both what it looks for and how it looks, InfEHR brings personalized diagnostics within reach, the investigators say.

In the study, InfEHR analyzed deidentified, privacy-protected electronic records from two hospital systems (Mount Sinai in New York and UC Irvine in California). The investigators turned each patient’s medical timeline—visits, lab tests, medications, vital signs—into a network that showed how events connected over time. The AI studied many of these networks to learn which combinations of clues tend to appear when a hidden condition is present.

With a small set of doctor-confirmed examples to calibrate it, the system checked whether it could correctly flag two real-world problems: newborns who develop sepsis despite negative blood cultures and patients who develop a kidney injury after surgery. Its performance in identifying patients with the diagnosis was compared with current clinical rules and validated across both hospitals. Notably, the system could also signal when the record lacked sufficient information, allowing it to respond “not sure” as a safety feature.

The study found that InfEHR can detect disease patterns that are invisible when examining isolated data. For neonatal sepsis without positive blood cultures—a rare, life-threatening condition—InfEHR was 12–16 times more likely to identify affected infants than current methods. For postoperative kidney injury, the system flagged at-risk patients 4–7 times more effectively. Importantly, InfEHR achieved this without needing large amounts of training data, learning directly from patient records and adapting across hospitals and populations.

“Traditional AI asks, ‘Does this patient resemble others with the disease?’ InfEHR takes a different approach: ‘Could this patient’s unique medical trajectory result from an underlying disease process?’ It’s the difference between simply matching patterns and uncovering causation,” says lead author Justin Kauffman, MS, Senior Data Scientist at the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine.

Importantly, in addition, InfEHR flags how confident it is in its predictions. Unlike other AI that may give a wrong answer with certainty, InfEHR knows when to say, ‘I don’t know’—a key safety feature for real-world clinical use, say the investigators.

The team is making the coding of InfEHR available to other researchers as it continues to study uses of the system. For example, the team will next explore how InfEHR could personalize treatment decisions by learning from clinical trial data and extending those insights to patients whose specific characteristics or symptoms were not fully represented in the original trials.

“Clinical trials often focus on specific populations, while doctors care for every patient,” Mr. Kauffman says. “Our probabilistic approach helps bridge that gap, making it easier for clinicians to see which research findings truly apply to the patient in front of them.”

The paper is titled “InfEHR: Clinical phenotype resolution through deep geometric learning on electronic health records.” The study’s authors, as listed in the journal, are Justin Kauffman, Emma Holmes, Akhil Vaid, Alexander W. Charney, Patricia Kovatch, Joshua Lampert, Ankit Sakhuja, Marinka Zitnik, Benjamin S. Glicksberg, Ira Hofer, and Girish N. Nadkarni.

This work was supported in part by the National Institutes of Health grant UL1TR004419, and the Clinical and Translational Science Awards grant UL1TR004419 from the National Center for Advancing Translational Sciences. Research reported in this publication was also supported by the Office of Research Infrastructure of the National Institutes of Health under awards S10OD026880 and S10OD030463.

For more Mount Sinai artificial intelligence news, visit: https://icahn.mssm.edu/about/artificial-intelligence.

Researchers found that ChatGPT 3.5, a form of artificial intelligence, fared worse than PharmD students in answering questions on therapeutics examinations that ensure students have the knowledge, skills, and critical thinking abilities to provide safe, effective and patient-centered care.

ChatGPT was less likely to correctly answer application-based questions (44%) compared with questions focused on recall of facts (80%). It also was less likely to answer case-based questions correctly (45%) compared with questions that weren’t focused on patient cases (74%). Overall, ChatGPT answered only 51% of the questions correctly.

The results provide additional insights into the uses and limitations of the technology and may also prove valuable in the development of pharmacy exam questions. The study findings appear in Currents in Pharmacy Teaching and Learning.

“AI has many potential uses in health care and education, and it’s not going away,” said Christopher Edwards, PharmD, an associate clinical professor of pharmacy practice and science. “One of the things we were hoping to answer with the study was if students wanted to use AI on an exam, how would they perform? I wanted to have data to show the students and tell them they can do well in the exams by studying hard and they don’t necessarily need these tools.”

A secondary goal was to find out what kinds of questions AI would struggle with. Coit College of Pharmacy Interim Dean Brian Erstad, PharmD, wasn’t surprised that ChatGPT did better with straightforward multiple choice and true-false questions and was less successful with application-based questions.

“The kinds of places where evidence is limited and judgment is required, which is often in a clinical setting, was where we found the technology somewhat lacking,” he said. “Ironically those are the kinds of questions clinicians are always facing.”

Edwards, Erstad, and Bernadette Cornelison, PharmD, an associate professor of pharmacy practice and science, evaluated answers to 210 questions from six exams in two pharmacotherapeutics courses that are part of the university’s Coit College of Pharmacy PharmD program.

The questions came from a first-year PharmD course focused on disorders related to nonprescription medications for heartburn, diarrhea, atopic dermatitis, cold and allergies. The other class was a second-year course that covered cardiology, neurology and critical care topics.

To compare the exam performances of pharmacy students and ChatGPT, they calculated mean composite scores as a measure of the ability to correctly answer questions. For ChatGPT, they added individual scores for each exam and divided by the number of exams. To figure out the mean composite score for the students, they divided the sum of the mean class performance on each exam by the number of exams. The mean composite score for six exams for ChatGPT was 53 compared to 82 for pharmacy students.

Educators, clinicians and others continue to debate the value of AI large language models, such as ChatGPT, in academic medicine. While such models will continue to play a range of roles in health care, pharmacy practice and other areas, many are concerned that relying too much on the technology could hamper the development of needed reasoning and critical thinking skills in students.

Both Erstad and Edwards acknowledged that in time, newer and more advanced technology may change these results.

Image: Brian Erstad, PharmD, is the interim dean and a professor at the R. Ken Coit College of Pharmacy.

View more Credit: Photo by Kris Hanning, U of A Office of Research and Partnerships