Last week I gave a talk to PhD students working on AI in Healthcare at University Hospital Essen (Germany). One comment I made which led to some discussion was that while AI in Healthcare models seem to be getting much better, we do not see much real-world impact on either patient outcomes or costs. I expand on the reasons for this below; I realise many other people have said similar things, but it is something I feel strongly about. I think similar issues may also affect non-medical uses of AI.

In my talk, I referred several times to the recent Stanford Arise report, which made comments such as “Model capability is accelerating, but evidence of real clinical impact remains limited”.

Reason 1: Real world healthcare is messy

Real-world healthcare is messy. Data is incomplete or incorrect, clinicians make mistakes, patients get confused, etc. Unfortunately, current AI models do not do well in messy uncertain contexts. A great example (which I have mentioned before) is Bean et al 2026, which shows that LLMs do a great job of answering patient queries if they are directly told the query or interact with a simulated user, but do a poor job if they interact with real people. This is because real people get confused, forget, misunderstand things, etc; a real-world patient-facing AI system needs to be able to cope with this.

Patient records are also often wrong or incomplete. Indeed, dealing with data quality issues was one of the biggest challenges in the Babytalk project we did back around 2010. However, as far as I can tell, the “look at amazing LLM performance” papers usually assume perfect data. The Arise report makes a related comment “[Frontier models] perform extremely well on complex reasoning tasks, yet break down when uncertainty, missing information, or changing context is introduced.”

Reason 2: Weak evaluation

Evaluation is AI for Healthcare is almost always done using benchmarks, test sets, LLM-as-Judge, etc (and also usually done in ways that ignore the messy real-world issues described above). Evaluation of real-world impact and utility is rare (paper). But benchmark (etc) evaluations are weak, especially in medicine where evidence-based medicine expects that interventions will be shown to be effective in (multiple) high-quality randomised controlled trials. My view is that benchmarks and test-sets can serve as screening tools, which filter out really bad approaches, but they cannot identify good approaches.

Manrai (2026) puts this well:

The medical AI field has become adept at building systems that perform well on an increasing array of benchmarks and tasks. The goal now is to build, and keep rebuilding, the rigorous and durable evidence base and instruments that inform clinicians and scientists about which medical AI tool to trust, and when.

Reason 3: Deploying AI in healthcare is difficult

It is difficult to deploy AI systems in healthcare contexts, especially for clinical usage (deployment for administrative tasks such as medical scribing is easier). I have already written a blog about this; very briefly, problems include lack of RCT evidence of effectiveness (mentioned above), poor return on investment (is AI a better investment than buying a new scanner?), and poor fit to workflows and indeed what clinicians and patients are looking for (researchers are fixated on decision support, while clinicians want help with admin tasks).

We also need to keep in mind that healthcare contexts are very different! A high-end US research hospital is a very different environment than an Indian primary health centre, and these need different solutions. Researchers love working with high-end hospitals, but I suspect that supporting Indian PHCs will do more to improve the health of large numbers of people.

Reason 4: Impact is disappointing

Last but not least, when healthcare AI is deployed on a large scale in the real world, its impact and effectiveness is often disappointing. A good example is medical scribes, which help doctors summarise consultations and otherwise write reports. This technology is being used by millions of doctors world-wide. A number of studies have been published of real-world impact, including by my PhD student Francesco Moramarco. In 2022, Francesco found a 9% decrease in report-writing time in real operational use, using a scribe based on a fine-tuned BART model. Our LLMs have greatly improved since 2022, but Lukac et al (2025) still report a 9.5% decrease for the best scribe they investigated. Even if modern LLMs make fewer mistakes, this may not have much impact on time savings as long as clinicians need to carefully check scribe-written reports.

Note that 9% is of marginal utility. If clinicians spend 10% of their time writing reports (which was the case for the clinicians that Francesco worked with), then using a scribe only saves 1% of the clinician’s time overall. We would probably get more time savings by reducing the number of useless admin emails that clinicians are expected to read.

A related point is that if our goal is large-scale impact, we need to change the healthcare system, for example by moving care out of hospitals and focusing on prevention instead of treatment (both of these are long-term goals of the UK health service). But I dont see much AI research which could lead to system change. Instead, most AI healthcare research focuses on things like helping clinicians diagnose scans; this kind of thing is not going to change the healthcare system.

Suggestions

If you want to work on AI in Healthcare and are worried by above, I suggest the following:

• (Impact) Choose an area where success could change things. For example, empowering patients to look after themselves has potential for high impact; I suspect this is less likely with improved medical scribes.
• (Deployment) Understand what clinicians and patients want, and also their concerns. We do a lot of this in my group (and not only in healthcare), using techniques such as focus groups, interviews, analysis of feedback on deployed apps, and analysis of social media comments.
• (Messy) Work with real users and data, not LLM simulations or synthetic data. Understand communication and data quality issues, and deal with them as best you can
• (Evaluation) Make your evaluations as realistic as possibe. Most researchers cannot do proper RCTs, but do your best to try your ideas out with real people in realistic contexts.

.