Doctors Are Worried About AI. They Use It Anyway. - The Atlantic

Every knowledge-based profession may one day reach the point when AI outperforms the human experts. In medicine, that day appeared to come in April. A group of primarily Harvard and Stanford researchers announced the results of a study that pitted ChatGPT against hundreds of physicians in a diagnostic obstacle course involving written medical mysteries and information from real-world patients. The bot had won, and the humans weren’t entirely happy about it.<br>“I get a little bit queasy about how some of these results might be used,” Adam Rodman, a lead author on the study, said at a press conference just ahead of its publication in the journal Science. The work had amounted to an academic exercise, he told reporters; as thorough as it may have been, it did not prove that ChatGPT or any other AI tool was ready to become a standard part of medical practice. His caution was in line with that of other experts, yet as Rodman knew, most people will ignore the warning. AI has already wormed its way into the U.S. health-care system, evidence and safeguards be damned.<br>Even as I was watching Rodman’s press conference, I got a message on my phone from the administrators at the medical center where I work as a pathologist. They’d emailed me to say that an “AI-powered clinical reasoning tool” was now available for me to use. This wasn’t the first time I’d gotten this sort of email; it wasn’t the second or third time either. In fact, I’ve lost count of how many generative-AI products have been rolled out to us in recent years, none of which has been approved for medical use by the FDA.<br>This enthusiasm feels unprecedented. Health care is typically among the last fields to adopt a new technology; I still use a pager, and I send faxes on a regular basis. (Younger readers can ask Claude to explain what these things are.) A tendency toward simple tech is in part a product of doctors’ safety-focused culture: We know that any ill-timed glitch has the potential to turn deadly. But these days, clinicians are allowed—encouraged, even—to run wild with the latest software, guided by a generic warning that “AI can make mistakes.”<br>Those mistakes can be consequential. Although Rodman’s research shows that generative AI can help diagnose rare diseases or make sense of unusual symptoms, a randomized trial that was published in NEJM AI just the week before found that intentionally erroneous output from an AI model can easily lead doctors astray. Nonprofessionals could be similarly misled. A recent study by Oxford scientists found that using AI did not significantly improve patients’ ability to diagnose themselves or others. Another one, led by researchers at Mount Sinai, suggested that chatbots may fail to alert users to potential medical emergencies.<br>Read: Drink whole milk, eat red meat, and use ChatGPT<br>Misdiagnosis is not the only concern. As AI permeates the health-care system, errors are cropping up in unexpected places. When I spoke with Rodman by phone after his press conference, he told me that he’d been surprised one day to find that his hospital, Beth Israel Deaconess Medical Center, had enlisted AI to draft messages to patients on his behalf—sometimes producing output for his review that he described as “completely absurd.” (Sarah Finlaw, a spokesperson for Beth Israel Lahey Health, told me that use of AI tools is voluntary and subject to hospital training and support. She also said that any output from AI tools must be approved by a physician.)<br>Part of the problem is that health-related AI products can be deployed without any vetting by officials at the FDA. If a software package that is intended for physicians is classified as a “clinical decision support tool,” and not a medical device, it usually avoids the agency’s oversight. To be counted in this category, an AI-powered app generally must rely on the existing medical literature, avoid analyzing medical scans or images, explain its reasoning, and leave diagnosis and treatment up to a physician.  Most of the generative-AI products that doctors use today seem to meet these criteria.<br>Consumer-wellness apps and devices may also bypass FDA review so long as they are intended for “maintaining or encouraging a healthy lifestyle” and not for diagnosing or treating specific conditions. With this in mind, Microsoft, OpenAI, Anthropic, and xAI all warn users that their health-related chatbots are not meant to provide medical care or issue diagnosis and treatment recommendations. In practice, though, the distinction isn’t always clear. Elon Musk encourages people to use his Grok chatbot to generate second medical opinions and interpretations of their X-ray and MRI images; a marketing video for ChatGPT Health shows the app reassuring people that their lab results are in a healthy range and encouraging them to continue taking cholesterol medication.<br>Most of these apps also invite users to connect their medical records and...