Skip to content

Ehud Reiter's Blog

Ehud's thoughts and observations about Natural Language Generation

  • Home
  • Blog Index
  • About
  • What is NLG
  • Publications
  • Resources
  • University
  • Contact
personal

Cycling through Southwest England

Jul 4, 2022Aug 5, 2022 ehudreiterLeave a comment

A travelogue about a recent bike trip. After two years of being limited in my holidays by Covid, it was great to finally be able to do some cycle touring again!

academics

Real-World Impact of Academic Research

Jun 16, 2022 ehudreiterLeave a comment

Society (and most funding agencies) want to see real-world benefits or “impact” from academic research. Of course not all research will have real-world impact, and impact may take years or decades to appear! I share some thoughts on types of impact, barriers to impact, and my personal experiences.

evaluation

Lets use error annotations to evaluate systems!

Jun 1, 2022 ehudreiter2 Comments

I am excited by the idea of using error annotation to evaluate NLG systems, where domain experts or other knowledgeable people mark up individual errors in generated texts. I think this is usually more meaningful and gives better insights that asking crowdworkers to rate or rank texts, which is how most human evaluations are currently done.

building NLG systems

NLG=Task+Data+Model/Alg+Eval

May 20, 2022May 20, 2022 ehudreiterLeave a comment

Progress in NLG requires understanding what users want, creating high quality data sets, building models and algorithms, and thoroughly evaluating systems. I remain disappointed that the research community seems fixated on building models and pays much less attention to user needs, datasets, and evaluation.

evaluation

We need more extrinsic (task) evaluation!

May 9, 2022 ehudreiter1 Comment

The most meaningful evaluation is when we test whether an NLG system actually achieves its communicative goal, eg helps people make better decisions or write documents faster. Unfortunately such “extrinsic” or “task” evaluation is rare in NLP in 2002, we need to see more such evaluations!

academics

Can I present my paper twice?

Apr 26, 2022 ehudreiterLeave a comment

I’ve come to realise that there is some confusion, especially amongst newcomers to NLP/AI, about when a research paper can be presented at two venues. I try to explain the rules and principles as I understand them.

evaluation

Why is ROUGE so popular?

Apr 10, 2022Apr 13, 2022 ehudreiter3 Comments

The ROUGE metric dominates evaluation of summarisation, and I do not understand why. I am not aware of good evidence that ROUGE predicts utility, and recent work by one of my students shows that character-level edit (Levenshtein) distance against a reference text is a better predictor of utility than ROUGE.

evaluation

Humans make mistakes too

Apr 3, 2022Feb 11, 2023 ehudreiter2 Comments

Some of my PhD students have recently looked at how many mistakes people (professionals, not Turkers) make when they do NLG-like tasks. The number of mistakes is considerably higher than we expected (although still much lower than the number of mistakes made by current neural NLG systems).

building NLG systems

Sports NLG: Commercial vs Academic Perspective

Mar 21, 2022 ehudreiter4 Comments

Both academic researchers and commercial NLG developers are interested in building NLG systems which describe sporting events. However, they care about different things. For example, many academics show little interest in use cases, domain knowledge, robustness, and high quality input data, all of which are very important to commercial NLG developers.

building NLG systems

Pragmatic correctness is a challenge for NLG

Mar 1, 2022Mar 3, 2022 ehudreiter4 Comments

NLG texts must be correct pragmatically as well as semantically. In particular, they must not contain statements which are contextually misleading even if they are literally true. We badly need better techniques for evaluating pragmatic accuracy as well as generating pragmatically correct texts.

Posts navigation

Older Posts
Newer posts
  • LinkedIn
  • Twitter

Top Posts & Pages

  • Does chatGPT make leaderboards less meaningful?
  • Can ChatGPT do Data-to-Text?
  • Bayesian vs Neural Networks
  • chatGPT: Great science, unclear commercials, hate the hype
  • Working in Universities vs Companies
  • Publications
  • How accurate do chatGPT texts need to be?
  • NLG and Explainable AI
  • "Will I Pass my PhD Viva"
  • What Should Academic NLP Researchers Focus on?
Blog at WordPress.com.
Ehud Reiter's Blog
Blog at WordPress.com.
  • Follow Following
    • Ehud Reiter's Blog
    • Join 80 other followers
    • Already have a WordPress.com account? Log in now.
    • Ehud Reiter's Blog
    • Customise
    • Follow Following
    • Sign up
    • Log in
    • Report this content
    • View site in Reader
    • Manage subscriptions
    • Collapse this bar
 

Loading Comments...