I’ve had several discussions over the past few months about evaluations that measure the real-word impact of NLG/NLP systems. In other words, instead of calculating metrics on test sets or asking human subjects whether they like what they see, we measure the impact on users when the system is deployed in the real world. I see lots of claims that LLMs will revolutionise the world in all sorts of ways, surely some of these claims should be backed up by experiments that measure real-world impact?

Unfortunately, real-world impact evaluations are rare in the NLP research literature; it does not seem to be part of our culture. Indeed, most of my discussions have been with researchers who work in other areas (such as medicine) where it is expected that high-quality evaluations will measure whether something works in the real world.

Of course commercial NLP companies have zillions of white papers and case studies which claim to demonstrate real-world impact, but these are marketing documents, not scientific experiments.

Anyways, I think we need at least some evaluations that scientifically measure real-world impact! Below I describe a few ways of doing this (there are of course others).

Randomised controlled trial

In medicine, the most rigorous and trusted type of evaluation is a randomised controlled clinical trial (RCT). I once did such an evaluation. We had developed an NLG system which produced personalised smoking cessation letters, and wanted to check whether the letters actually helped smokers quit. So we set up an RCT where 2500 smokers were sent either our NLG letter or control material. We waited six months, identified the individuals who had stopped smoking, and measured whether there were more such individuals in the NLG-letter group than the control groups (unfortunately, one of the control groups had more quitters than the NLG group, so our system was not effective).

Large numbers of RCTs have been done in medicine, and I’ve also seen them used in other fields such as psychology and even economics. But they remain very rare in NLP, even in NLP for clinical applications. One of PhD students, Mengxuan Sun, who is working on NLP in cancer care, did a survey of papers that applied NLP in this area, looking at papers published up to 2022; she found lots of impressive technology and interesting use cases, but nothing remotely resembling an RCT.

Process efficiency

For a number of years I worked in applications of NLG in the oil industry, and during this period I attended several engineering conferences in this area. They had numerous papers where an engineer applied an innovation to an oil platform, refinery, etc, and then reported how this innovation impacted production, efficiency, downtime, etc. We could do the same in NLG, especially for document production tasks.

Another PhD student, Francesco Moramarco, in fact did this for an NLG system which summarised doctor patient consultations; these summaries were edited by doctors before being saved in the patient record. Francesco analysed data from 5000 consultations where the system was used with real patients and compared these to consultations which did not use the system. He showed that the system saved doctors some time (although perhaps not as much as hoped) without diminishing quality; indeed reports produced by this process (doctors post-editing NLG summaries) had fewer errors than manually-written reports.

It is very common for LLMs to be used in “human-in-loop” contexts, and in principle it should be relatively straightforward to do similar studies in other such use cases; but I have seen very few such studies in the research literature.

A/B Testing

It is common in IT for companies to evaluate new versions of websites and other services by A/B Testing, ie providing the new website to some customers and the old website to others, and measuring whether this has an impact on their behaviour. I’ve never done this myself, but it seems like a sensible way to evaluate the impact of many NLP systems. Indeed, I suspect some NLP companies do this, ie test new versions of models using A/B testing, but I’ve only seen a handful of papers in the research literature that evaluate systems using A/B testing.

Retrospectives

Finally, it would be useful to see retrospectives, which look back on a product or system and analyse successes and failures in real-world usage. I know lots of organisations do this internally, and see this as a valuable source of lessons for the future, but again I’ve rarely seen this kind of thing published.

One excellent example of a published retrospective in NLP is Strickland’s retrospective analysis of IBM Watson in healthcare (blog). This is full of very useful and valuable insights which I suspect generalise to other attempts to use in AI and healthcare, such as the difficulty of keeping ML models up-to-date with the latest medical findings.

I’d love to see more such retospectives!

Let me know if I’ve missed something!

There are thousands of NLP research papers published every year, and I’ve only read a handful of them, so I probably have missed papers that evaluate real-world impact of NLP. Please let me know if you’ve seen a good paper along these lines! Provided that its a scientific paper, I’m not interested in commercial white papers or case studies.

Also do let me know if you are interested in carrying out an evaluation of real-world impact, I am also happy to give advice if this is appropriate.