The paradox of AI in healthcare is that while benchmarks show soaring (and sometimes superhuman) performance, in the real world AI is not actually helping people much. This is partially because real-world healthcare is messy, AI/NLP people do weak evaluations, deploying AI systems in real-world healthcare is challenging, and impact is limited even when systems are deployed.

25 years ago I proposed personal health assistants as a grand challenge for computer science. LLMs have brought this vision closer to reality, but many challenges remain. These include understanding requirements, adapting to individual users, showing effectiveness in RCTs, and running on cheap phones with limited Internet access.

There is very limited data on harms to real patients from using AI health chatbots. The limited data we have from incident reports, clinical trials with patients, and data from health providers suggests that bots are usually safe, but can cause harm in a few cases. More data is badly needed!

I am excited by the idea of using AI to help people manage ilness and health conditions. This isnt very sexy, but I think there is real potential to improve health outcomes and quality of life.

I review some data on usage of AI in healthcare, and conclude that the most common uses in 2025 are probably (A) giving personalised health information to patients and (B) helping clinicians write documents. We’ve worked on these topics at Aberdeen, but most researchers focus on AI for decision support, which is not widely used.

My student Adarsa Sivaprasad is looking into what questions users of an AI prediction model actually have, and how these should be answered. Amongst other things, users seem to have more questions about what information a model considers than about how a model works.