Skip to content

Ehud Reiter's Blog

Ehud's thoughts and observations about Natural Language Generation

  • Home
  • Blog Index
  • About
  • What is NLG
  • Publications
  • Resources
  • University
  • Contact

Tag: accuracy

academics

Academic NLG should not fixate on end-to-end neural

Dec 1, 2020Dec 1, 2020 ehudreiter1 Comment

I was shocked when a PhD student recently told me that he thought he had to focus on end-to-end neural approaches, because this dominates the conferences he wants to publish in. I’m all for research in end-to-end neural, but fixating on this to the exclusion of everything else is a mistake. Especially since end-to-end neural approaches do not currently work very well.

evaluation

Evaluating Accuracy

Nov 17, 2020Dec 3, 2020 ehudreiter9 Comments

Craig Thomson and I will present a paper at INLG on a methodology for evaluating the accuracy of generated texts, based on asking human annotators to mark up factual errors in a text. This is not cheap, but I think it is the most robust and reliable approach to measuring accuracy.

Uncategorized

Accuracy Errors Go Beyond Getting Facts Wrong

Apr 27, 2020 ehudreiter9 Comments

Accuracy errors in NLG texts go far beyond simple factual mistakes, for example they also include misleading use of words and incorrect context/discourse inferences. All of these types of errors are unacceptable in most data-to-text NLG use cases.

Uncategorized

Shared Task on Evaluating Accuracy?

Feb 18, 2020Feb 1, 2021 ehudreiter1 Comment

We’re thinking of organising a shared task on evaluating the accuracy of texts produced by NLG systems. Comments welcome, also let me know if you might participate.

Uncategorized

Generated Texts Must Be Accurate!

Sep 26, 2019Sep 26, 2019 ehudreiter12 Comments

I’ve been shocked by the fact that many neural NLG researchers dont seem to care that their systems produce texts which contain many factual mistakes and hallucinations. NLG users expect accurate texts, and will not use systems which produce inaccurate texts, not matter how well the texts are written,

  • LinkedIn
  • Twitter

Top Posts & Pages

  • We need to understand what users want!
  • Bayesian vs Neural Networks
  • Blog Index
  • "Will I Pass my PhD Viva"
  • Hallucination in Neural NLG
  • Publications
  • NLG vs Templates: Levels of Sophistication in Generating Text
  • Small differences in BLEU are meaningless
  • Natural Language Generation and Machine Learning
  • Accuracy, Fluency, and Utility
Blog at WordPress.com.
  • Follow Following
    • Ehud Reiter's Blog
    • Join 75 other followers
    • Already have a WordPress.com account? Log in now.
    • Ehud Reiter's Blog
    • Customise
    • Follow Following
    • Sign up
    • Log in
    • Report this content
    • View site in Reader
    • Manage subscriptions
    • Collapse this bar