A few weeks ago I was chatting to one of our PhD students, who had presented a paper at ACL 2018 (which he was justifiably very proud of). He asked me about future publications, eg should he publish in ACL again or go for other high-prestige conferences such as NAACL or EMNLP. I told him he should consider publishing in a journal such as TACL, Computational Lingiustics, or Natural Language Engineering, and he was very surprised; he had never previously really thought about publishing in a journal instead of a conference.
Advantages of Publishing in Journals
I suspect a lot of students (and indeed other researchers) in NLP have similar attitudes; they are focused on conference publication, and don’t consider journals. Which is a real shame, because I think it is much *better* to publish in journals, for the following reasons:
Better reviewing process: Journal authors can properly interact with their reviewers, and improve their paper based on reviewer’s suggestions and concerns. Reviewing has absolutely improved many of my papers (see previous blog), and I like to think that my comments have improved many of the papers I have reviewed. This can happen with conferences as well, since usually authors can submit revised versions after seeing reviews, but in my experience its much less common. Partially because there isnt much time for this (usually people only have a few weeks to prepare final versions), but also because many conference paper authors don’t in fact significantly update their papers, which in turn discourages reviewers like me from making suggestions. Certainly when I review a journal paper, my mindset is “how can this paper be improved”; whereas when I review a conference paper, my mindset is more “accept or reject”.
Flexibility over timing and size: Conference papers are submitted on a rigid schedule and need to follow rigid size constraints. Journal papers, in contrast, can be submitted at any time, and often have some flexibility about size as well. So if I am publishing in a journal, I dont need to wait for a suitable conference deadline, I can submit when I’m ready. And if I need a few more pages, its often (not always) possible to arrange this.
No travel: If you are presenting at a conference, you need to physically go to the conference. While for many young researchers this is a perk, for someone like me (with a many family and professional responsibilities) it can be a real hassle. The need to travel is also unfair to researchers who dont have much funding, such as most researchers from poorer countries, and more generally people who are not very good at playing the “funding game”.
Disadvantages of Publishing in Journals
Are there advantages of publishing in conferences? In the past, one factor was that journals were slow in publishing papers. I think this is less true in 2018 than it was in the past, though. In particular, TACL usually gets papers out in 6-9 months (from first submission to formal publication), which is not much longer than conferences (and of course TACL papers can be submitted at any time, you dont need to wait for a conference deadline). Computational Linguistics is getting better as well. My most recent paper there (A Structured Review of the Validity of BLEU) (which benefitted immensely from reviewers comments and suggestions!) only took about a year from initial submission to formal publication (and it was available 9 months after submission under the journal’s Early Access tab).
Another issue (which the above-mentioned student told me) is the perception that conference papers are better from a CV perspective than journal papers. I dont think this is true; certainly when I look at CV’s I put more value on a good journal paper than a good conference paper. Having said this, I realise that TACL in particular does not yet have a journal impact factor, which causes problems in some CV-related contexts. However, if you’re worried about this, there are plenty of NLP and AI journals (including Computational Linguistics) which do have impact factors.
Incidentally, a week before my above-mentioned discussion with the NLP student, I attended a faculty meeting about publications, and people there from other disciplines were astounded that in Computer Science, conference papers are regarded as serious publications! CS is very much the outlier in this regard, almost all other fields expect serious research results to be published in journals.
In all honesty, I suspect many people prefer conferences because its easier to publish low-quality research in conferences. There are a lot of really bad papers in ACL conferences (I wrote a blog about one, which unfortunately was not atypical). And its not just ACL. I recently asked my MSc students to choose a paper from IJCAI 2018 and evaluate its quality, and I was pretty shocked by what they found; most of the papers they looked at had major problems, and some were scientifically worthless. You get weak papers in journals as well, but they are less common, and its very rare to get a paper in a good journal which is completely worthless.
And this should not be surprising to anyone who is familiar with the reviewing process. Conference reviewing is essentially becoming a lottery, where outcome may bear little relationship to quality. In part because the skyrocketing number of submissions to AI and NLP venues means that reviewing is done by people who are (A) under a lot of time pressure (B) to review many papers (C) many of which are in areas they know little about. Whereas journal reviewing is still (mostly) of high quality, at least for good journals.
I think conferences and workshops are great venues for presenting speculative ideas, work in progress, etc. But journals are the best way to publish solid research results!
12 thoughts on “Publish in Journals!”
Thanks for writing this piece! I fully agree with the reviewing process for conference vs journal. However, I can’t think of a realistic solution to address the needs of lots and lots of young researchers getting into the field, and their needs/requirements to publish as a part of their degree, to get a job etc. You publish, say 5 conference papers, vs 1-2 journal papers – recruiters view the 5 more than the 1-2. It is the truth while applying for industry positions which most of these graduates will go for, as there are neither that many academic positions, nor is there that sort of money in academia. (One may say: we should not work for money, but a harsh reality for some of us is that money is needed and it needs to be earned periodically to survive).
I appreciate what you are saying, and indeed another young researcher made a similar comment to me. One approach might be to publish your best result in a journal, and the rest in conferences. That would keep up your numbers, but also give you one high-quality journal paper. And having such a paper can help in some recruitment contexts (of course depends on the company and job).
In other words, conf vs journals is not “all or nothing”. What I’d ike is people to at least consider the journal route for their best work, instead of automatically publishing everything in conferences.
Thank you for raising the issues. I agree that journal reviewing usually makes the paper better. One thing that discourages me is that I’ve received many bad journal reviews, meaning that reviewers ask still didn’t really get their heads around the content. But as journal reviewers they have a lot of power to make you do many revisions, extra experiments, etc. One reviewer made us redraw all figures in the paper, not for the content, simply the looks! Editors often do a poor job too: they simply pass revisions and comments back and forth, without selecting which comments are reasonable and important.
Many people have commented on the issues you raise, often humourously.
About unreasonable reviewers: Cartoon http://jasonya.com/wp/your-manuscript-on-peer-review/
Hilarious fake letter to editor: http://eloquentscience.com/wp-content/uploads/2012/02/Frustrated_author.pdf
The ACM ran a symposium on conference reviewing. Thomas Anderson on “conference reviewing considered harmful” https://homes.cs.washington.edu/~tom/support/confreview.pdf . About journal reviewing he asks “how do we get there from here”? That’s my question too.
You might also want to check out Moshe Vardi’s several editorials on conference reviewing, he has been raising this topic in the CACM. In one editorial he says that CS being the only conference-based discipline reminds him of the man driving on the wrong side of the road who was wondering “what is wrong with all the other drivers today”.
Many thanks for your comment and links. You can certainly get poor reviews from journals as well as conferences. But journal reviewing also has the “upside” potential to make a paper much better. Many of my journal papers have been massively improved by reviewers comments; my conf papers have also benefited from reviews, but its been smaller incremental improvements. Likewise as a reviewer I like to think that many of the journal papers I reviewed have benefited from my comments; I cannot say this about conf papers (especially since many conf authors ignore reviewers comments once their paper is accepted, which in turn demotivates me from suggesting improvements when I review conf papers).
About editors, there are of course good and bad journal editors as well as conf (area) chairs. The difference is that a good journal editor can and will get involved, especially when reviews are biased or otherwise inappropriate. Its much harder for an area chair at a good conference to put effort into addressing reviewing issues for one paper, because he/she is dealing with large numbers of papers and reviewers on a very short timescale.
And I absolutely agree that we need to improve peer review! There are no shortage of good suggestions (including the ones in the links you cite), but change is slow. Which perhaps is inevitable.
Hi Ehud, thank you for the reply!
Journals require thorough evaluations which tend to be expensive. Conferences can be more feasible for researchers with limited resources. I have been in a situation where I did some robot work with a collaborator who could use their lab to carry out some experiments for me. I did not have any hardware to do experiments myself. The evaluation was not thorough enough for a journal but it did make it in a conference. The conference publication is very valuable to me (and also to the co-authors), to show credibility, for example, for future funding applications. Apart from robotics, I have seen some very thorough evaluations in top journals in other areas which use Amazon Turk and would be very expensive. There is also the time taken to carry out the evaluation which is another resource that not everybody might have available.
The conference paper is a cheaper “product” for sure, but it is sometimes the only affordable option.
In the case I described my department was able to fund me to go to the conference, but could not have funded the resources for a journal quality evaluation.