Reviewing for big NLP conferences has changed drastically since 1990, when 11 senior researchers reviewed all ACL submissions. Perhaps our expectations about conference papers also need to change, and become more similar to expectations in other scientific fields.
The Covid crisis gives us a chance to rethink and change our conference culture. I would like to see fewer large international conferences, and also have these focus on discussion and interaction rather than on oral presentations.
A few comments on how I review papers (what I actually do, not what I am supposed to do), and associated advice for authors.
I went to my first developers conference last week and was impressed, not least by the sensible attitude towards deep learning and other trendy AI technology.
Ehud Reiter's Blog 