Science is an iterative process. The greatest achievements in any scientific field come from generations of investigators building on the works of their intellectual ancestors. In order to achieve maximum potential, progress in science must be shared.
The primary mechanism that has arisen to share scientific scholarship is through journals. Peer review has emerged as a crucial part of the publication process. Peer review is barrier to entry and a quality control measure to improve the reliability of research before it is widely disseminated.
Many visitors at this site have participated in peer review. For others, I suspect peer review is a bit of a black box. As someone who been on both sides of peer review many times, I hope to describe and demystify the process. My interactions with peer review are within a tiny medical subspecialty, but I suspect my general description of peer review applies across disciplines.
Peer review. Why?
Scientific journals have limited space and resource to publish papers. The number of articles submitted for consideration, in most cases, far exceeds these limits. It is in a journal’s best interest to select the most meritorious papers to fill their pages. Journals lack sufficient in-house manpower to critically review every manuscript. There is limited breadth, based on the sheer number of submissions. There is limited depth-based need for highly qualified experts in sometimes quite esoteric fields. Peer review is a method of outsourcing review to colleagues matching the needs for each submission. The editors use these reviews to inform their decisions about which articles to eject and to improve the quality those which they publish.
Peer review from an author’s perspective
I complete my research and am excited to share it with the world. I first need to decide which journal I would like to have publish my article. In most disciplines there are many choices. The decision about where to submit is based on many factors. Journals specialise to some degree about the types of research they publish. If my article is a clinical trial comparing strategies for the management of age related macular degeneration, it would be inappropriate to submit it to a psychiatry journal. It would be nearly as inappropriate to send it to an ophthalmology journal that specialises in pediatric ophthalmology.
There is also a hierarchy of journals based on reputation. In general, authors are motivated to aim for more prestigious journals. Getting published in an elite journal validates the quality of a researcher’s work and ensures broad readership. However, one must be realistic. It is foolish to send a low-impact, preliminary study to a highly-selective, prestigious journal. It is simple supply and demand. Elite journals tend to have very strict standards. They receive lots of quality manuscripts, and most of them are rejected.
Once I choose where to submit my paper, I create and format a manuscript in compliance with the standards of the journal. I complete the required accompanying documents and upload my manuscript. I immediately get a polite acknowledgment from the editorial office, thanking me for my submission. Then, I wait.
The manuscript is first reviewed by the editorial staff, who will usually prescreen the manuscript to make sure the subject matter is appropriate for their journal, and to look for any glaring deficiencies that would make it unfit for publication. Many manuscripts get rejected and returned to the author prior to peer review. Although it is frustrating to have a paper rejected in this way, it serves a valuable purpose: if fatal flaws are discovered prior to peer review, time and peer reviewer resources are not wasted. Equally, I do not have to wait weeks or months for a paper to undergo full peer review only to be rejected – I can go about the business of correcting the deficiencies and resubmitting to an appropriate journal.
If the paper survives preliminary review from the editorial staff, it is sent for formal peer review. The editors will select two or more reviewers with expertise in the subject of the manuscript. Each reviewer independently dissects the manuscript and assesses its strengths and weaknesses, writing comments and suggesting edits.
Once the editorial staff has compiled the reviewers’ assessments and comments, I get a reply from the editor. The reply will include comments from the reviewers, as well as comments from the editor. There are several possibilities:
- My paper is rejected. The editor explains the rationale for the rejection, usually referring to reviewers’ comments. It may be that the research is not up to the standards of the journal. The research may be fine, but not high enough impact or priority to justify publication.
- My paper is not accepted, but the editor is open to sending the paper for peer review once again if major edits are made. This option may be offered if the editor deems the work meritorious but has significant deficiencies requiring attention. The needed modifications will be explained. I may or may not decide to resubmit the paper to this journal.
- My paper is provisionally accepted. The editor will require me to respond to the comments of the reviewers and edit the manuscript where necessary. If the replies and edits are satisfactory, the paper will be published.
- My paper is accepted. Unconditional acceptance almost never happens (maybe that’s just me).
If the paper is provisionally accepted, I write a revised manuscript, taking into consideration the comments of the editor and the reviewers. I return the revised manuscript and add a cover letter replete with gratuitous compliments to the editor and to the reviewers for their “thoughtful reviews” and “insightful suggestions.” I respond to the comments and questions from the reviewers and explain how they have been addressed in the revised manuscript. The editor will determine if my responses and edits are appropriate. If so, the paper will be accepted for publication.
Once accepted, copy editors prepare the article for publication. They will usually propose some edits to conform with editorial conventions and improve the grammar and syntax. They may have some questions about ambiguous language in the manuscript. Figures and tables are created and formatted where needed. As the author, I will be given a chance to answer questions and to reject or accept the editor’s modifications.
The article then goes into a queue for publication in print, digitally, or both.
If my paper is rejected, I need to make an assessment. Was the research weak? Was the paper badly written? Did I aim too high in my target journal? The comments for the reviewers should be helpful in sorting out the issues. I may choose to do some additional research or analyses to strengthen the science. Usually I will choose another journal, edit the manuscript, and try again.
Authors’ lament
As a human endeavor there is a spectrum in quality of peer review. Most experienced authors remember manuscripts they feel were rejected based on unfair reviews. They can probably recite a litany of idiotic comments from clueless reviewers. Yes, I have had a few. However, in my experience, most reviewers are diligent, thoughtful, and strive to give a critical but fair review. In most cases, they point out legitimate deficiencies in the manuscript. If an expert peer reviewer has a difficult time understanding something I have tried to communicate, the average reader will have even greater difficulty. The process almost always improves the quality of the ultimate publication.
Peer Review from the reviewer’s point of view.
I’m minding my own business and an email shows up in my inbox. It is an invitation to review a manuscript. The email includes the title of the manuscript, and usually the abstract. I am requested to reply within a couple of days with a decision to accept or decline the invitation.
There are many reasons I might decline an invitation to review a paper. I may be asked to review a paper on a subject I lack sufficient expertise. The invitation may arrive at a time when I lack sufficient time to do a thorough review within the requested timeline. I could have a conflict of interest.
If I accept, there is a deadline (usually a couple of weeks) to complete the review. I will be provided a link to manuscript.
I will be expected to review the paper in great depth and provide a detailed commentary on the strengths and weaknesses. Among the questions I will assess: Is the research question well-defined? Is adequate background information provided? Are the methods adequately described? Are they appropriate for the research? Are the results clearly presented? Are they complete? Are the statistical methods appropriate? Are the results put into the proper context? Are the conclusions justified by the results? Are the weaknesses and limitations of the research acknowledged? Are the references adequate and cited appropriately?
I will be asked to construct a set of comments and questions that will be shared with the authors. This might include requests for clarification or expansion of certain sections of the paper. I might recommend that the authors report additional analyses to further clarify their results. I might suggest that figures or tables be added, omitted, or combined. I might comment that the discussion is too long, too short, or overly speculative.
The editor will usually ask me a few specific questions about the quality, originality, and potential impact of the research. There is an opportunity to provide confidential comments to the editor that will not be shared with the authors. This could include sensitive information, such as if some kind of misconduct is suspected.
They will also usually ask for recommendation of whether to accept, provisionally accept, or reject the paper.
The editor considers the opinions of all the peer reviewers before sending a decision letter to the author. The letter will include comments from the reviewers, comments from the editor, and the disposition to accept or reject the paper. Decisions to reject are usually final, but if an author feels that the paper has been unfairly reviewed, they may appeal the decision.
Peer-reviewer’s lament
Quality, conscientious peer-review is very time consuming. Peer-reviewers are not paid. Most of us consider peer review as a valuable service we endure as ’payback’ to keep the wheels turning and promote publication of quality research.
Reading quality original research and providing constructive feedback to help authors to communicate their creation most effectively is a noble and rewarding endeavor. Sometimes, however, we are asked to review really bad papers. The research may be ill-conceived, poorly executed, badly reported, misinterpreted, and unoriginal. Sometimes the grammar and syntax are dreadful. It can be frustrating to spend hours poring over an incoherent unsalvageable manuscript; then crafting diplomatic, constructive comments to the authors, all the while feeling you have put more thought into this project than they did.
Controversies in peer review
Peer review is a time consuming, labour-intensive process. Editors struggle to find sufficient numbers of qualified, willing reviewers. The process slows down the scientific community’s access to contemporary research by weeks, months, or even years. Any given paper might be arbitrarily assigned to a negligent or under-qualified reviewer. Flawed papers get published. Good science gets rejected. So why bother? Despite the flaws and inefficiencies, there is broad consensus that peer review is a valuable, even necessary step in the dissemination of scientific discoveries.
Peer review is only as good as the process a journal exercises and quality of the peer reviewers they select. There are numerous examples of ’peer-reviewed’ journals accepting blatantly unmeritorious papers. Open access journals, in particular, have been singled out as sometimes employing questionable peer review standards. Open access journals have a business model that differs from traditional journals. Traditional journals rely on subscriptions and advertising for revenue. These journals have an incentive to maintain a reputation for publishing top-notch research in order to broaden their subscribers and attract more advertisers. The downside is that these journals are very expensive. Articles may not be easily accessible to those without a subscription or access to a medical library.
Open access journals are open access: no subscription required. They are published online only – anyone with an internet connection can read them. They earn revenue by charging authors for publishing material in their journal. This model provides less incentive for quality and greater incentive for quantity, which is not a motivation for rigorous peer-review standards. Many open access journals have successfully navigated this landscape. Most notably, PLOS ONE (Public Library of Science) is considered a reputable, open access journal. On the other end of the spectrum, there has been a proliferation of open access journals with low standards and low readership that prey on authors who are desperate to get their research published and willing to pay for the privilege. These journals are nominally peer reviewed, but publication practices are often quite promiscuous. These are sometimes known as “predatory journals.” In one of the most extreme (and hilarious) example, an author repeated a profane 7 word request (I encourage you to click the link, but it is NSFW) to be removed from a mailing list in a format that resembled a scientific manuscript. If the manuscript had received even a cursory review by a human, it would have been immediately dismissed. It was accepted for publication.
There are genuine weaknesses and criticism of peer review. Journals are experimenting with modifications. For the most part, reviewers do their work anonymously. This arrangement is intended to enable reviewers to be honest and not fearful of reprisal from angry authors. Some journals are trying open peer review. In this model, peer reviewers are identified by name. This is intended to promote greater transparency in the process and greater accountability on the part of reviewers. On the other end of the spectrum, there are some calls for the abolition of pre-publication peer review as we know it.
I believe there is a purpose and place for the imperfect process we call peer review. I am hopeful that through imagination and innovation the weaknesses and limitations will be improved.
