There is very limited data on harms to real patients from using AI health chatbots. The limited data we have from incident reports, clinical trials with patients, and data from health providers suggests that bots are usually safe, but can cause harm in a few cases. More data is badly needed!
Quantitative comparisons of different LLMs are not very interesting in research papers, because the LLMs in question will probably be out of date by the time the paper is published. However looking for behaviour which is shared by several LLMs is definitely interesting and worthwhile.
ACL/ARR have rules and guidelines for how papers are written. Unfortunately many authors (and reviewers) ignore these, which makes their papers harder to read and less useful. Please follow the rules!
A group who is reading my book sent me many questions, some of which we discussed in a call last week. I thought I would share the questions and my responses.
Most semantic evaluation of LLMs focuses on accuracy and hallucination. These are very important, but it is also important to look at completeness and omission; does the generated text include all of the key information which the user needs to know? Omissions are a huge problem in medical NLG, and in other NLG tasks as well.
The most exciting and rewarding moments of my research career were when I discovered something new and exciting about NLG, language, etc. I describe a few of these “Eureka” moments. I hope my readers also have them; when I reflect back on my career, these moments and insights are what I remember best, much more so than getting papers or proposals accepted.
I am excited by the idea of using AI to help people manage ilness and health conditions. This isnt very sexy, but I think there is real potential to improve health outcomes and quality of life.
I hope to retire soon, and many people are asking about my plans. Basically I want to do lots of travel, say involved in academia, and perhaps do some writing.
I strongly recommend that researchers do “sanity checks” on data, model outputs, and evaluation results, looking for anomalies. This can help detect data errors, model cheating, software bugs, and other flaws which distort experiments.
LLMs often “cheat” on benchmarks via data contamination and reward hacking. Unfortunately, this problem seems to be getting worse, perhaps because of perverse incentives. If we want to genuinely and meaningfully evaluate LLMs, we need to move beyond benchmarks and start measuring real-world impact.
Ehud Reiter's Blog 