I dont like academic leaderboards. Poor scientific techniques, poor data, and poor evaluation means leaderboard results may not be worth much. I also suspect that the community’s fixation on leaderboards also means less research on important topics that do not fit the leaderboard model, such as understanding user requirements.
Quality assurance processes for academic research, notably peer review by unpaid volunteers, are very lightweight and miss many problems. Better quality assurance processes would require more resources and efforts, but would result in more trustworthy papers.
I was very impressed by a recent talk about the power of simple white-box models in tasks such as medical diagnosis. I’d love to see more work on simple models in NLP and NLG!
Thge most populat datasets used in summarisation (CNN/DailyMail and XSum) do not actually contain summaries. I find this worrying. Surely the best way to make make progress on summarisation is to use actual summarisation datasets, even if these are less convenient from a “leaderboard” perspective.
I’m considering writing a book on NLG (a mere 22 years after my last one), and would welcome feedback from the community on this project.
Language is diverse, and different syntax, vocabulary, document structures, etc are used in different domains and genres. NLG developers and researchers need to keep this in mind if they are trying to develop generic NLG components.
I am excited by the idea of using a neural language model to improve the output of rule/template NLG. Many academics probably regard this as a boring use of LMs (see my previous blog), but I think it could be very useful in many real world applications.
There is lots of excitement and hype about “gee whiz” uses of language models in NLG, such as generating stories from prompts. However, I suspect there maybe more real-world value in using language models for more mundane tasks such as quality assurance.
We can build much better NLG systems if we understand what users want the systems to do! This may sound trite, but there is very little research in the academic community in understanding user needs and requirements, which is a shame and indeed lost opportunity.
I was very happy to win an INLG Test of Time award for my paper “An Architecture for Data-to-Text Systems”, so I thought I’d write a few comments on it.
Ehud Reiter's Blog 