My Guidelines for Evaluating AI Systems
Ehud’s guidelines for evaluating AI systems: keep it simple, keep it ethical, be careful, do proper stats, and be skeptical
Author: ehudreiter
Managing Research Projects is Painful but Necessary
Managing research projects is not much fun, but it is important. I have seen many research projects which did not reach their potential or even failed completely, because of poor management.
Testing Multiple Hypotheses
The NLP/AI community needs to do a better job of dealing with multiple hypotheses, otherwise a lot of our results will be garbage.
Embedding Machine Learning in a Rules-Based NLG System
I think we should use rules to make simple high-value decisions, and learning to make complex low-value decisions, within an architecture where ML decision makers are embedded in a rules-based framework.
Non-Experts Struggle with Information Graphics
Imformation graphics work well for domain experts. but they are not nearly as useful for junior professionals. And the “man in the street” may struggle to understand anything more complex than a simple bar chart.
My Father Takes Me to Mexico
My father died recently, and I spoke at his funeral about a trip we took together to Baja California (Mexico) when I was 11 years old.
Does Quality Matter in Training Data?
I am concerned that some people seem to ignore quality issues in training data.
Research Ethics of A/B Testing
What are the ethical issues when academics do A/B testing?
Is BLEU valid? First observations and concerns
The first phase of my systematic review of BLEU shows that BLEU-human correlations are all over the place, and that none of the studies in my review have correlated BLEU with real-world utility or user satisfaction.
How do Users React to NLG?
Some obervations on how people react to NLG systems (which is a very different issue than scientific evaluation).
Ehud Reiter's Blog 