I was recently elected to the board of ACL SIGGEN (Association for Computational Linguistics, Special Interest Group on Generation), which is the closest thing the NLG community has to a professional body.  My fellow board members then asked me to be chair of SIGGEN.   So I thought I’d write a few thoughts on what I want SIGGEN to do.

How Has NLG Changed since early 2000s?

I was on the SIGGEN board in the late 1990s and early 2000s (I dont remember the exact dates).   The NLG community has certainly changed quite a bit since then!  Its much bigger, like most types of AI and NLP.   150 people attended INLG 2018; INLG’s in the early 2000’s would be lucky to attract half this number.  There are also a growing number of “specialist” NLG events (which was unheard of 15 years ago).  NLG also has a rapidly growing commercial presence in 2019 (Arria and its competitors, plus many NLG projects within large companies), and there are articles about NLG in the business press and by technology forecasters such as Forester.  In 2000 there was only one specialist NLG company, CoGenTex (which was struggling), and very few people in the business community had heard of NLG.

Technology-wise, a lot of people in 2019 are working on neural and deep learning techniques, which did not exist 15 years ago.   Evaluation is also much more important now, as is releasing data sets and open-source software.  Data-to-text has really come of age, which has led to exciting research on the relationship between analytics and NLG, and also on how text and graphics can be used together to communicate information.  Language grounding has also emerged as a serious topic, which is long overdue.

So overall NLG is much healthier in 2019 than in 2004!   But there are some problems, and hopefully SIGGEN can help with these.  These include

• Poor scientific quality, especially with deep learning approaches to NLG.
• Insufficient interaction with people outwith AI
• Growing commercial marketing/hype make it difficult to find trustworthy accurate information

Poor Scientific Quality

I have been horrified by the scientific quality of a lot of the deep learning NLG work I have seen. To be honest, much of what I have seen is ML at its worst.  Researchers who have little understanding of NLG come up with inappropriate datasets, meaningless evaluation metrics, and terrible baselines, and then happily publish streams of papers about how some tweak on an LSTM gives slightly better performance on this pointless data, as measured by meaningless evaluation metric, compared to the terrible baseline.

Fortunately I think the situation is getting better, and researchers with a good understanding of both NLG and deep learning are investigating the challenges in using DL in NLG, including hallucination, evaluation, and lack of large amounts of training data.

I think SIGGEN can help by encouraging and supporting events and forums that bring together NLG and DL experts to discuss these challenges.   I also wonder if SIGGEN can enourage reviewers at NLG events to insist that papers have proper evaluation, appropriate datasets, etc.

Insufficient Interaction With Outside World

The NLG community has always been somewhat insular, and this needs to change.  I remember being really angry around 15 years ago when I encouraged a medical researcher who had done some work on medical NLG to submit a paper to an INLG conference, and it got rejected because his paper was written like a medical paper, which is a very different style from the usual NLP paper.   This kind of thing is really bad for the community; we need to welcome interested outsiders and encourage them to participate in our events!   We also need to make it easier for interested outsiders to learn about and experiment with NLG.

I think SIGGEN can help with both of these.  One idea I have (which is probably controversial!) is to encourage events to be selective about full papers (need proper evaluation, etc), but not very selective with posters.  So pretty much anyone who has something to say and wants to interact with the NLG community can present a poster about his/her work and ideas.

About information, SIGGEN has a website, which unfortunately is largely out of date.  I’d like to update the website with links to high quality sites with NLG information, tools, resources, etc.  And keep it up-to-date (which is probably more of a challenge!).

Commercial/Marketing Hype

If you did a Google search on NLG 15 years ago, you would have seen links to academic sites which generally were of high quality.  If you do a Google search on Natural Language Generation in 2019, you will probably see the Wikipedia page and then several pages from commercial companies.   I think the NLG Wikipedia page is OK (disclosure: I wrote most of it), but it can be a struggle to keep it free of marketing and indeed commercial attacks by one company against another.  The commercial pages are a mixed bag: sometimes interesting but usually biased.

Maybe I’m being idealistic, but wouldnt it be great if high-quality, up-to-date, trusted and impartial SIGGEN pages came out on top of the search results?  And wouldnt it be great if a trusted SIGGEN web page or Twitter account was the place people turned to for news about NLG?

Representing the NLG Community

The above vision is why I ran for the SIGGEN board.   But now that my fellow board members have asked me to be Chair of SIGGEN, I also may find myself representing the NLG community.    I’m not entirely sure what this means in practice, but I do promise to do my best to be fair, objective, and represent the community as a whole!