Its reviewing season again for academic papers, and I’ve seen a number of comments from people about how reviewing ought to happen.    I thought I’d write a few comments about how I actually review papers, and what this means for authors.  My reviewing is certainly not ideal, but may perhaps reflect how other people review papers as well.

Is it Interesting, Useful, or Exciting to Me?

Sometimes when I read a paper, I saw “wow, this is really useful to me”, “fascinating stuff”, or “makes me think and see things in a new and different light”.   If so, I will be in a “positive” mood when I review the paper, and more likely to generous when considering potential flaws.  On the other hand, if I read a paper and think “this is so boring and useless to me”, this will put me in a negative mood when I review the paper, and make me more likely to be harsh about problems.   Note that I have rejected papers which “wowed” me because I decide they had too many problems and were not yet ready for publication; and I have accepted papers that bored me because they were solid and I thought would be of interest to others.  However, if a paper is on the borderline (as so many are), my personal reaction to it is pretty important to my decision.

This is not the way reviewing is supposed to work, because of course what interests and excites me is different from what interests and excites other people.  For example, a paper on a psychologically inspired model of lexical choice is likely to interest and excite me, but bore someone whose focus is deep learning.  The reverse would be true of a paper on adjusting deep learning models which focused on math and made no reference to language or users.   What this means is that my reviewing outcomes are different from other people, which in turn means that the reviewing process is very noisy.   Ie, whether a paper (or research proposal) is accepted depends largely on who is chosen to review it.  Which is something that many people have observed and even scientifically studied (eg, Pier et al 2018).

Reviewing is not supposed to work this way, reviewers are supposed to leave behind their personal interests and review papers on behalf of the community.  And perhaps some people do this.  But I am certainly influenced by my interests, and I suspect others are as well.

Advice for authors: If a paper is rejected, and you think the rejection is unfair, you can resubmit it elsewhere.  If your paper was rejected because it was really bad, this won’t work.  But if it was rejected because of “bad luck” in reviewers, then resubmission makes sense, because perhaps another set of reviewers may like it.  Certainly several of my most-cited papers were rejected when I first submitted then, but accepted when I resubmitted them elsewhere.

Is the Paper Readable?

Some papers are really hard to read.  When I was younger, I sometimes would spend a lot of time trying to decode what the authors were trying to do, and then write a review based on this understanding.  But now that I am older and perhaps more cynical, my attitude is that if I cant understand the paper after two attempts, then it should not be accepted; the authors should rewrite the paper to make it clear what they are doing, and then submit the revised version.

In principle, we are supposed to review based on science, not presentation.  And I dont reject papers that have some presentation problems if I can still figure them out.  But my feeling now is that a paper which is really difficult to read is not going to be very useful to the research community, because no one is going to read it.

Advice for authors: Make sure your paper is readable by reviewers who are generally knowledgeable in the field but may not be experts in your specific area.   In particular, you should ask colleagues to review papers for readability before you submit them.  Its sometimes difficult for authors themselves to identify readability problems, because they of course understand the work perfectly, and hence may not realise when something is poorly explained.

Does the Paper Include a Solid Evaluation?

In my view, the evaluation is usually the most important part of a paper.  If a paper has a poor evaluation, it is not a research contribution.   If a paper has a solid evaluation, this will usually teach us something, no matter how “disappointing” the outcome.

I have written numerous blogs on how to evaluate NLG systems (eg, Types of NLG Evaluation: Which is Right for Me?), so I wont go into detail here.  However I will say that I expect evaluation results to be reported honestly as well as done correctly.  If the evaluation produces a negative result (ie, your idea didnt work), please say so, dont try to obfuscate with lots of commentary and secondary analyses.    Similarly if you are essentially replicating an experiment, just say so.  I personally am happy to accept papers which present negative results and/or replicate previous experiments.   On the other hand, I get seriously annoyed by papers that try to obfuscate evaluation results because they were not what the authors wanted to see.

Advice for authors: Do a proper evaluation, which is carefully thought out, carefully executed, and properly written up.  Dont just run stuff through a few metrics and/or do a “quick&sloppy” Mechanical Turk study.  Also please dont limit your “results” to generic statements such as “evaluation  results were very positive”.

Conference or Journal?

I review journal papers differently from conference papers, because journal reviewing is more interactive.   In particular, when I review for a journal, I can say “accept the paper if the authors change X, Y,and Z”, and then check the revised submission to make sure that the authors have indeed addressed X, Y, and Z.   This option doesnt exist for conferences; I can recommend changes, but the authors are free to ignore them (and often do exactly this).

Which means that if I review for a journal, I tend to write long reviews with many suggestions and comments as well as required changes; this is because I feel that I am interacting with authors who will listen to me.  Whereas if I review for a conference, I write shorter reviews with fewer suggestions and advice, since authors are likely to ignore anything I say.

Advice for authors: If you want detailed and constructive reviews, and to interact with reviewers, you should submit your paper to a journal, not a conference.  In general, as an author I prefer submitting major papers to journals instead of conferences: better reviewing process, I can submit at a time of my choosing, and I dont need to worry about whether I am able to travel at a specific conference date.