I am very happy to see that the MT community is adopting the annotation-based MQM protocol as a gold-standard evalution technique. Having such a gold standard both strengthens evaluation and also supports exciting new research in evaluation.
A lot of experiments in NLP are neither rigorous nor replicable, and some NLP researchers don’t seem to care. Which is depressing… But I do see interest in incrementally improving experimental rigour and replicability, I strongly encourage people to try to do a bit better, even if they cant fix everything.
I’ve written a new book on NLG which was published in late October 2024. It tries to present a broad perspective on NLG, including requirements, evaluation, and use cases as well as technology. Perspective is similar to my blog in many ways, and I hope blog readers find it useful.
Deployed software systems need to be maintained as bugs emerge and the domain and user needs evolve; this perhaps is especially challenging for systems based on LLMs. Unfortunately little is known about maintaining NLG systems,
In NLG we focus on quantitative evaluation, but qualitative techniques can also be used. Quantatitive hypothesis testing is essential, but its also really useful to ask people what they think of an NLG system in an open-ended way.
The UK government wants to reform the UK health system by digitisation, shifting care to communities, and focusing on prevention. I think there is a lot of potential for AI to help with this, if AI/Medicine researchers become less fixated on using AI to improve diagnoses in hospitals.
In early Sept I ran a one-day class on evaluation. I summarise what I did in this class and give links to my presentations, in case this is useful to other people.
Sometimes the latest technology is *not* appropriate for an NLG task. I saw this very strongly in the late 2010s with LSTMs (which do not work well for data-to-text), and continue to see this in 2024 (GPT4 is not always the best approach). Both researchers and developers need to be open-minded about alternative approaches.
I’m now part-way towards retirement, and working fewer hours (for less money) while have more time for trips and other personal activities. A few people have asked about this, so I thought I’d explain in a blog.
AI has many promising applications in healthcare, but adoption of AI in healthcare is very slow. One message from a recent workshop I attended is that it would help if AI researchers had a better understanding of requirements of the health sector, including evaluation, challenges, and business cases.
Ehud Reiter's Blog 