A well-written peer review can push a research article from being simply promising to publishable. The goal of the peer review, a critical inspection of an article manuscript prior to its publication, is to make suggestions on how to improve the article. But how to do that? We asked journal editors how to do an excellent peer review.
The detailed knowledge in a specific subfield provided by expert reviewers is invaluable when deciding whether to proceed with a submission or not. This article is an attempt to summarize what constitutes an excellent peer review. We reached out to Professor Jennifer Gibbs at the University of California, Santa Barbara, who is an experienced editor, to pinpoint the elements of peer review excellence. Jennifer Gibbs is currently an editor of Communication Research and serves on the editorial boards of a number of other top-tier communication journals.
Despite the fact that the system of peer review has been criticized, and interesting alternatives such as open peer review are emerging, blind peer review is still the predominant way to ensure high-quality research publications. And researchers like the system of peer review. A survey among over 4,000 researchers around the globe concluded that the peer review process is highly regarded by the vast majority of researchers and considered by most to be essential to the communication of scholarly research. In fact, 90 per cent of authors thought that peer review made their most recently published paper better (Mulligan et al., 2013).
While most scholars seem to appreciate the feedback provided by a thorough reviewer, the feelings towards the task of reviewing itself seems to be more complex. On the one hand, one is aware of the value that a good review brings to the process of journal article publication and acknowledges that reviewing is an opportunity to stay informed about the latest state of the art in one’s field. On the other hand, reviewing a journal submission is time consuming, and the compensation is often of minimal, symbolic, or abstract in character, such as “making a contribution to your field”. This might be especially true for young scholars who are new to peer reviewing and are uncertain about what is expected from them. This article provides some concrete advice from an editor’s point of view.
Constructive
When writing your review, no matter how passionate you are about the subject matter or how upset you might feel that important works are not cited (often your own), try your very best to be constructive. To simply point out what is wrong with the paper is a review half done. If too confrontational, the author might miss your point because they will be busy defending their work against a hostile attack. Professor Gibbs offers the following concrete recipe for a constructive review:
– A constructive review follows the “compliment sandwich” structure of starting with positive feedback (what the paper does well), then getting to the “meat” of the critique (which makes up the bulk of the review), and ending on an encouraging note (wishing the authors well with their continued research, even if the decision is negative). It engages with ideas and suggests constructive paths forward, rather than simply ripping the paper apart. It avoids personal attacks and insults.
A constructive review is also one that departs from the submitted manuscript, rather than some ideal paper that the reviewer might have written themselves if addressing the subject matter. “Often, reviewers will critique a paper because the authors did not do the study the way the reviewer would have”, according to Gibbs.
– A great review assesses the manuscript based on its own merits and goals. This includes assessment of methods. It is not helpful when reviewers critique a qualitative study using post-positivist criteria (e.g., the sample is not generalizable or controlled enough), or say that the authors should have done an experiment instead of a survey, because that’s what the reviewer would have done.
Organized
Another virtue of a good peer review is a thought-through structure.
An organized review has a better chance of getting the message through to both editor and author, increasing the chances of improving the submission. Editors can help by summarizing the main points when passing on the review to the author, but such bullet points can seldom compensate entirely for a messy review.
During her years as a journal editor, Gibbs noticed that reviewers often take a chronological approach and list their concerns in order of appearance in the manuscript.
– There are several problems with this approach. First, it fails to synthesize individual concerns into larger issues. Second, it fails to distinguish between major and minor points, since each point is presented as equally important.
Rather than just going through the manuscript page by page, Gibbs recommends starting with an introductory paragraph that briefly summarizes the manuscript and gives an overall perspective.
– It is especially helpful if this paragraph ends with a sentence that sums up the major concerns in one sentence to help frame the review. For example: “Despite these positives, I have major concerns about the lack of conceptual clarity, representativeness of the sample, and the way in which key variables were measured”. This helps the editor orient quickly to the review and identify the main concerns, as well as the overall tone of the review.
Continuing, Gibbs also finds it very helpful if reviewers take the time to include subheadings to highlight the main themes of the review, preferably to provide integrative themes as headings (e.g., Rationale and Framing, Theoretical Development, Conceptual Clarity, Analysis).
– It takes more work to synthesize critiques into common categories. But highlighting the higher-level concerns is incredibly helpful for the editor, who is trying to make sense of your review.
Critical
As an editor of Nordicom Review, I have noticed that reviewers are sometimes reluctant to recommend a rejection, despite doing a good job of pointing out the major revisions needed to make a submission publishable.
Although Gibbs does point out that the recommendation should be in sync with the overall tone of the paper (see below), the reluctancy to recommend a rejection is perhaps more prominent for editors of smaller journals. Nordicom Review communicates personally with each reviewer, and as an organization, Nordicom has also consciously positioned itself at the center of efforts to strengthen the community of media and communication researchers in the Nordic countries. Both things might make reviewers feel that they are expected to recommend resubmission rather than rejection. This is a misconception likely to do more harm than good.
As a reviewer, you are not invited to simply praise the paper. You are invited by editors to identify weaknesses in the manuscript and discover blind spots, missing references, unfinished or limping argumentation, etc.
In a fascinating paper, D’Andrea and O’Dwyer (2017) write: ”We find that the biggest hazard to the quality of published literature is not selfish rejection of high-quality manuscripts but indifferent acceptance of low-quality ones”. In other words, if you think the paper is bad and that so many revisions are needed that making them requires a work effort corresponding to writing an entire article – recommend rejection. Also, it might help to remember that when recommending rejection, you are not saying that the research in itself is bad, simply that the paper is not in the shape it needs to be for a smooth publication process.
The Don’ts
Lastly, there are a few things that editors (and probably authors too) dread. First, don’t provide a recommendation that does not match the overall tone of the review. Second, avoid pettiness. As an editor, waiting several weeks for a review and then receiving a list of minor issues such as spelling errors or font changes for headings (yes, it happens) is very disappointing.
To sum up, a review should be constructive in tone, organized in structure and honest. Good luck in reviewing your next paper!
References
D’Andrea, R., & O’Dwyer, J. P. (2017). Can editors save peer review from peer reviewers? PLoS ONE, 12(10), e0186111. https://doi.org/10.1371/journal.pone.0186111
Mulligan, A., Hall, L., & Raphael, E. (2013). Peer review in a changing world: An international study measuring the attitudes of researchers. Journal of the American Society for Information Science and Technology, 64(1), 132–161.
Suggested Events
Participate in the Peer Review Week
Photos: Pimthida, Jason Truscott, Flickr