Thge most populat datasets used in summarisation (CNN/DailyMail and XSum) do not actually contain summaries. I find this worrying. Surely the best way to make make progress on summarisation is to use actual summarisation datasets, even if these are less convenient from a “leaderboard” perspective.

Language is diverse, and different syntax, vocabulary, document structures, etc are used in different domains and genres. NLG developers and researchers need to keep this in mind if they are trying to develop generic NLG components.

I am excited by the idea of using a neural language model to improve the output of rule/template NLG. Many academics probably regard this as a boring use of LMs (see my previous blog), but I think it could be very useful in many real world applications.

There is lots of excitement and hype about “gee whiz” uses of language models in NLG, such as generating stories from prompts. However, I suspect there maybe more real-world value in using language models for more mundane tasks such as quality assurance.

We can build much better NLG systems if we understand what users want the systems to do! This may sound trite, but there is very little research in the academic community in understanding user needs and requirements, which is a shame and indeed lost opportunity.

Progress in NLG requires understanding what users want, creating high quality data sets, building models and algorithms, and thoroughly evaluating systems. I remain disappointed that the research community seems fixated on building models and pays much less attention to user needs, datasets, and evaluation.

Both academic researchers and commercial NLG developers are interested in building NLG systems which describe sporting events. However, they care about different things. For example, many academics show little interest in use cases, domain knowledge, robustness, and high quality input data, all of which are very important to commercial NLG developers.

NLG texts must be correct pragmatically as well as semantically. In particular, they must not contain statements which are contextually misleading even if they are literally true. We badly need better techniques for evaluating pragmatic accuracy as well as generating pragmatically correct texts.

There is a lot of uninformed criticism of rule-based NLG in academic papers. In this blog I explain at a very high level how such systems work and what some of the main challenges are in building them.

The fundamental challenges of building useful data-to-text NLG systems are the same regardless of whether we build systems with rules or transformers. We need to understand where NLG is useful, choose good content to communicate, robustly deal with edge cases, allow users to configure and control the system, and evaluate properly. I’d like to see more research on these fundamental issues, regardless of technology used.