Most health AI tools being used right now have never had to prove they actually help patients

That is the uncomfortable truth at the center of the JAMA Summit Report on AI in Health and Health Care - and if you work anywhere near digital health, regulation, or policy, this paper deserves your full attention.

The FDA only sees a fraction of the picture. Over 1,200 AI tools have received FDA clearance, mostly in medical imaging. But the 21st Century Cures Act explicitly exempts a large swath of AI from FDA oversight entirely: scheduling tools, billing software, certain clinical decision support systems, EHR tools, and most general wellness apps. That is the majority of AI touching patients every day, operating without any federal requirement to demonstrate it works.

Even for tools that do go through FDA clearance, the agency doesn't necessarily require proof of improved clinical outcomes. A tool can be cleared without ever demonstrating that patient health actually gets better. Safety and effectiveness are not the same standard, and clearance does not guarantee the latter.

The direct-to-consumer market makes this worse. There are now over 350,000 mobile health apps, in a market worth more than $70 billion annually. Most are governed primarily by the FTC, whose mandate covers privacy and deceptive advertising - not whether the health advice an app dispenses actually works. Some have been shown to give guidance contrary to clinical guidelines or to fail users during mental health crises, with little consequence.

Business operations AI sits in an even deeper regulatory vacuum. Tools that optimize hospital scheduling, manage revenue cycles, or drive prior authorization denials have virtually no framework requiring evaluation of their effects on patient health. Many US physicians believe AI-driven prior authorization is already causing widespread harm. There is not a single peer-reviewed study that has actually measured this. The former FDA Commissioner put it plainly: he did not believe there was a single health system in the United States capable of validating an AI algorithm once it had been put into clinical care.

The Summit's proposed path forward rests on four things. First, moving away from the traditional linear model - where development, evaluation, regulatory review, and monitoring happen sequentially in separate silos - toward genuine multistakeholder engagement across the entire product lifecycle. Second, developing evaluation frameworks that measure health outcomes, not just safety signals and process compliance. Third, building a national data infrastructure capable of generating real-world evidence at scale, along the lines of what the FDA's Sentinel program does for drug safety. Fourth, getting the incentive structures right, because market forces alone will not produce the oversight this moment requires. The report points to the HITECH Act as a working model: a relatively modest federal investment of $35 billion drove EHR adoption to 97% of health systems within a decade.

The contrast with Europe is worth noting. The EU has moved toward a comprehensive AI regulatory framework. The US has moved in the opposite direction. The gap between how fast AI is being adopted in clinical settings and our collective ability to know whether it is safe, effective, and equitable is widening.

AI will reshape every part of healthcare delivery. The question that should concern regulators, health systems, and policymakers is not whether that disruption is coming - it is whether the disruption will improve health for everyone, or primarily for those with the resources to navigate a system that, so far, has asked very little of the tools it is quietly embedding into care.

Full report: https://jamanetwork.com/journals/jama/fullarticle/2840175