Close to half of the AI devices authorized for use by the U.S. Food and Drug Administration (FDA) are trained without using clinical data from actual patients, according to new findings by an international team of researchers.
Artificial intelligence offers many promising benefits to the medical field, from creating and updating patient charts as needed to diagnosing health issues with unprecedented accuracy. However, implementing these tools is becoming a challenge. The adoption of AI in medicine has been met with skepticism, mainly due to concerns about patient privacy, bias, and device accuracy.
A multi-institutional team of researchers from the University of North Carolina (UNC) School of Medicine, Duke University, Ally Bank, Oxford University, Columbia University, and the University of Miami is addressing these concerns. Led by Sammy Chouffani El Fassi, an MD candidate at UNC, and Gail E. Henderson, Ph.D., a professor at UNC, the team conducted a thorough analysis of clinical validation data for over 500 AI medical devices.
Their findings, published in Nature Medicine, revealed that nearly half of the AI devices authorized by the U.S. Food and Drug Administration (FDA) lacked reported clinical data from actual patients.
“Although AI device manufacturers boast of the credibility of their technology with FDA authorization, clearance does not mean that the devices have been properly evaluated for clinical effectiveness using real patient data,” Chouffani El Fassi said in a recent statement. “With these findings, we hope to encourage the FDA and industry to boost the credibility of device authorization by conducting clinical validation studies on these technologies and making the results of such studies publicly available.”
The Rise of AI Use in Medicine
For the past decade, the use of AI in medicine and healthcare has been steadily increasing. Part of this demand comes from the shortage of medical professionals and the rise of telehealth, where patients are diagnosed remotely instead of going in person. The COVID-19 pandemic significantly increased telehealth visits, driving companies to invest in virtual software to host these visits. AI has been used to help manage this software, update patient forms, and assist with scheduling and other logistics.
AI’s use in medicine can not only save time but could also help save significant costs in the long term. According to a 2020 paper: “It is estimated that AI applications can cut annual US healthcare costs by $150 billion in 2026.”
AI-based medical devices, in particular, can streamline processes, reduce the need for expensive diagnostic tests, and minimize human error, translating into substantial cost savings for healthcare systems. However, the cost-effectiveness of these AI tools hinges on their accuracy and reliability in real-world clinical settings.
Training with Fake Patient Data
As the number of FDA authorized AI-based medical devices has risen from two to 69 since 2016, these approved devices should have been trained on real patient data to insure the best accuracy in their applications.
Yet, the study found that 43% (or 226) of these devices (our of 521 approved devices) lacked published clinical validation data, and some even relied on “phantom images” rather than real patient data.
“A lot of the devices that came out after 2016 were created new, or maybe they were similar to a product that already was on the market,” Henderson explained in a recent statement. “Using these hundreds of devices in this database, we wanted to determine what it really means for an AI medical device to be FDA-authorized.”
Lack of Reinforcement
To make matters worse, the team found that in the September 2023 FDA guidance document, the most recent version, there is no distinction between different types of clinical validation studies for recommendations to device manufacturers, meaning that there’s weaker reinforcement in making sure these devices are trained on real patient data.
In the study, the researchers advocated for clearer distinctions between different types of clinical validation—retrospective, prospective, and randomized controlled trials—within FDA guidelines. They hope their findings will influence FDA regulatory decisions and inspire global research efforts to improve the safety and effectiveness of AI medical technologies.
“We shared our findings with directors at the FDA who oversee medical device regulation, and we expect our work will inform their regulatory decision-making,” said Chouffani El Fassi. “We also hope that our publication will inspire researchers and universities globally to conduct clinical validation studies on medical AI to improve the safety and effectiveness of these technologies. We’re looking forward to the positive impact this project will have on patient care at a large scale.”
Kenna Hughes-Castleberry is the Science Communicator at JILA (a world-leading physics research institute) and a science writer at The Debrief. Follow and connect with her on X or contact her via email at kenna@thedebrief.org