How to Validate Metrics
My advice on how to perform a high-quality validation study, which assesses whether a metric (such as BLEU) correlates well with human evaluations.
Tag: evaluation
Why doesnt BLEU work for NLG?
BLEU works much better for MT systems and NLG systems. In this blog I present some speculations as to why this is the case.
BLEU in Different Languages: Dont use it for German
My structured survey of BLEU suggests that BLEU-human correlations are worse in German than in many other languages. But there are many caveats, so we need to be cautious in interpreting this result.
BLEU-Human Correlation is Increasing: What does this Mean?
The correlation between BLEU and human evaluations of MT systems seems to be increasing over time. Since BLEU has not changed, how is this possible, and what does it mean?
Many Papers on Machine Learning in NLP are Scientifically Dubious
In response to a previous blog, many people expressed concerns to me about the quality of many papers they saw on ML in NLP. I summarise some of these concerns, which are worrying.
Learning does not require evaluation metrics
I was recently asked if machine learning requires evaluation metrics. The answer is no, and the fact that people are asking such questions suggests that some newcomers to the field may have a limited perspective on NLP research methodology.
Please Use Two-Tailed P Values!
If you are writing a scientific paper which presents statistics, please use two-tailed p values unless you **really** know what you are doing.
