If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Various types of bias and confounding have been described in the biomedical literature that can affect a study before, during, or after the intervention has been delivered. The peer review process can also introduce bias. A compelling ethical and moral rationale necessitates improving the peer review process. A double-blind peer review system is supported on equipoise and fair-play principles. Triple- and quadruple-blind systems have also been described but are not commonly used. The open peer review system introduces “Skin in the Game” heuristic principles for both authors and reviewers and has a small favorable effect on the quality of published reports. In this exposition, we present, on the basis of a comprehensive literature search of PubMed from its inception until October 20, 2017, various possible mechanisms by which the peer review process can distort research results, and we discuss the evidence supporting different strategies that may mitigate this bias. It is time to improve the quality, transparency, and accountability of the peer review system.
Various types of bias and confounding have been described in the biomedical literature.
•
Bias attributable to the peer review process is less recognized.
•
Different peer review systems exist, and some may introduce less bias than others.
•
Experimental studies are needed to identify the least biased peer review approach.
•
We explore mechanisms of peer review bias and offer potential solutions.
•
A compelling ethical and moral rationale necessitates improving the quality, transparency, and accountability of the peer review process.
The first documented description of a peer review process to regulate the medical profession was in a book titled Ethics of the Physician (Adab al-abīb) written by the Syrian physician Ishāq bin Ali Al-Rahawi (854-931 A.D.) from Al Raha in northern Syria.
This book indicates that one of the duties of a physician is to write duplicate notes for their patients. When the patient had recovered or had died, one of these notes was to be examined by a senior medical council to assess whether malpractice had taken place according to the standards of that time. Peer review to adjudicate scientific publications can be traced back to 1752, when the Royal Society of London created a committee to assess the papers submitted to their journal Philosophical Transactions. The objective was to assist the editor in choosing appropriately themed articles and not to strictly assess their quality.
The scientific peer review process in its actual form began in the United States in the early 1900s, and Science, The Journal of the American Medical Association (JAMA), and American Practitioner were the first journals to implement it. This process was greatly facilitated by the introduction of photocopiers in 1958, which allowed the circulation of several copies of submitted manuscripts to external reviewers without losing the original documents.
Peer review is perceived as contributing to science in 2 ways: assisting the board of editors to accept or reject a paper and helping to improve submitted manuscripts by eliminating major flaws and gaps.
Considering the important effect of the peer review process on the integrity of research, we conducted a narrative critical review of studies that evaluated peer review bias in biomedical research.
Methods
We searched PubMed from its inception until October 20, 2017, using the Boolean MeSH terms peer review AND bias. We limited the search to articles written in English having the Boolean MeSH terms in the title and abstract. In addition, we manually searched the reference lists of relevant publications for additional articles.
Peer Review as a Source of Bias
Scientific experiments in general, and biomedical research specifically, are subject to bias and confounding. Domains for the different types of bias and confounding have been established as to their role before, during, and after the intervention is delivered.
Dedicated tools have been developed to evaluate the bias risk in randomized controlled trials (RCTs), observational studies, diagnostic studies, and others. The role of risk of bias domains has been studied and its effect has been estimated. For example, in randomized trials with inadequate allocation concealment or lack of double-blinding, odds ratios were exaggerated by 41% and 17%, respectively.
Conceptually, the peer review process can lead to distortion of the results from the viewpoint of the evidence user, akin to bias. Peer review bias can be defined as a violation of impartiality in the evaluation of a submission.
We propose that this transgression of neutrality standards can affect the dissemination of research (ie, denying publication, which would be a form of publication bias), or modify the spin and communication of findings (affecting the impact and implementation of research), and potentially introduce other unknown biases. This concern was demonstrated in an illustrative study in the journal Brain and Behavior Sciences by Peters and Ceci in 1982.
Peer review bias is difficult to detect and quantify. One can even argue that such distortion may not fit the classical definition of bias as a unidirectional systematic deviation from truth. On occasions, it may. Peer reviewers can ask authors to delete outcomes, combine outcomes, modify analyses, perform post hoc subgroup analyses, and perform other actions that can, in fact, introduce classical bias. A proposed framework to illustrate a possible role for the peer review process to introduce bias is depicted in the Figure.
FigureProposed framework demonstrating how the peer review process can introduce various types of bias.
The effect of peer reviewers that is mediated by the classical publication bias is intuitive. Nonsignificant results can be denied publication by peer reviewers and may never get published or may go through multiple journals, leading to delay and eventual publication in journals with limited visibility or a low impact factor or in a language other than English.
Akin to how traditional bias can occur before, during, or after research, peer review bias can be viewed in 3 phases. The “before” phase includes the editor's appraisal of articles before sending them to peer review and making judgments based on the country of origin, institution, or a renowned author. Decisions by the editorial office regarding which peer reviewers to seek can also introduce bias. The “during” phase includes the dogma of reviewers, differences in the proposed paradigm with the authors, and preconceived notions by peer reviewers. The “after” phase includes stalling and delaying a publication or perhaps writing an editorial that might mitigate/negate/temper the findings of the study.
The effect of authors' conflicts of interest on studies' evaluations is well known and has been reported by Chaudhry et al.
The significance of reviewers' conflicts is increasingly recognized, and most editors now ask their reviewers to report their conflicts of interest before reviewing or to contact the editors in case of doubt.
In single-blind peer review, the reviewers know the identity of the authors but not vice versa. Single-blind peer review has remained the norm in clinical medicine and biomedical journals.
The primary advantage of single-blind peer review is that reviewers are free from authors' influence, which enables them to be straightforward in their comments and appraisals without fear of conflicts.
Although knowledge of authors' identities and their institutions is advantageous to famous authors and authors from prestigious institutions, it is also detrimental to junior investigators.
In double-blind peer review, the identities of both authors and reviewers are kept hidden from each other. Double-blind peer review has been used in imaging, nursing, and humanities journals.
Double-blind peer review is considered more objective and fair, while still offering all the advantages of single-blind peer review. Although it is plausible that double-blind peer review would protect the authors from reviewers' bias, this premise requires substantiation with stronger evidence. Certain aspects of reviewers' abuse persist whether in a single- or double-blind peer review system, such as plagiarism of nonpublished data for personal benefit or deliberate delay of publication to usurp the authors.
A reviewer may take a longer time in a double-blind system because he or she may spend a lot of time uncovering the author's identity instead of focusing on proper review of the manuscript.
True blinding could be difficult to accomplish. Reviewers can identify the authors successfully in 25% to 50% of cases in biomedical or social science journals.
Common reasons for blinding failure are reviewer familiarity with authors' work and when the authors inadvertently reveal their identity in the manuscript.
Phrases such as “we have shown that” should be replaced by “previous studies have shown that.” Authors are advised to avoid the use of self-citations and citations that are “in press” because such practices indicate that the work is done by the same authors. Furthermore, authors should remove their initials from the manuscript and remove names of institutions from images and drawings.
Although it could be argued that the double-blind peer review process adds burden to the editorial staff workload and may increase publishing cost, in the era of electronic submission methods and publication checklists, some of these theoretical concerns can be addressed.
Triple-Blind Peer Review
In this model, in addition to blinding the reviewers' identity, the authors' identity is also hidden from the editors during the submission process, and a deidentification code is assigned. Thereafter, the editorial board guides the manuscript without knowledge of the authors' names or institutions. All communications occur through the journal's website without divulging any author-related identifying features. This approach is implemented by numerous academic journals in the fields of philosophy (eg, Philosophy and Phenomenological Research, The British Journal for the Philosophy of Science) and quality improvement (eg, BMJ Quality & Safety) and has reportedly been trialed recently by the journal Molecular Cell.
The steps of double-blind and triple-blind peer review processes are preserved and augmented by hiding the identity of the handling editor. The anonymous editor is appointed by the editorial board committee, and his or her identity remains confidential throughout. One attempt to reflect a quadruple-blind peer review process has been demonstrated in the submission policy of Ethics.
In an open peer review system, both authors and reviewers know the identity of each other. Whereas The British Medical Journal (BMJ) succeeded in moving to an open peer review system nearly 2 decades ago, Nature and PLOS Medicine have had less successful attempts because of nonengagement of the authors and reviewers.
Supporters of this system believe that it increases transparency, encourages reviewers to be constructive in their comments, and could prevent plagiarism.
The open peer review system is associated with a higher refusal rate from reviewers and an increased time to write the reviews. Some journals ask the authors to suggest reviewers, but these reviewers may be more favorable than those nominated by the editors. Furthermore, this approach may be conducive to scientific fraud.
Many journals are implementing the open peer review system despite the lack of supporting evidence. Seventy journals from BioMed Central currently adopt different models of open peer review.
It is difficult to answer this question with certitude. Each system (single-blind, double-blind, triple-blind, quadruple-blind, or open) has advantages and challenges.
Furthermore, the double-blind peer review system seems less biased at face value. Burdens on editorial staff can be reduced by a checklist system and software developments (eg, having separate entries for authors' names, institutions, and the body of the manuscript that yields automatically an unidentified manuscript for circulation).
A systematic review and meta-analysis of RCTs that evaluated the impact of different interventions on the quality of peer review has been published recently.
There was no statistically significant difference in the quality of peer review report and recommendations for rejection between single- and double-blind systems. However, the absence of a statistically significant difference should not be interpreted as a lack of blinding benefit because this body of evidence warrants low certainty (a high level of heterogeneity, a small number of RCTs, and insufficient data to conduct important subgroup analyses). Open peer review had a small favorable effect on the quality of peer review report and decreased recommendations for rejection (Table).
Revelations of fabricated data in the New England Journal of Medicine and other prestigious journals have shown that peer review was unable to detect fraud.
Make per review scientific: thirty years on from the first congress on peer review, Drummond Rennie reflects on the improvements brought about by research into the process--and calls for more.
The latter example of scientific misconduct took place over several years and affected at least 30 publications in peer-reviewed journals. Fraud in most cases is the work of individuals, not a team, and it is thought that close supervision by senior scientists participating in clinical trials could preclude such regrettable events.
The peer review process is far from perfect. Dr Richard Smith, former editor of BMJ, described the single-blind peer review system in an editorial published in 1999 as “slow, expensive, highly subjective, prone to bias, easily abused, and poor at detecting gross defects.”
and the most important question in this regard is not to abandon it but how to improve it.
Peer review involves several groups: authors, who want to publish their work for career promotion or to procure funding; editors, who are on the lookout for novel and sound research; reviewers, who try to find sufficient time in their busy schedules and often without remuneration or academic credit; and publishers, who compete with other rivals in a fast-changing field.
The ultimate stakeholder in this process is the patient, whose care and even survival may depend on the sound flow of the peer review process. These groups have to be engaged collectively to improve the peer review process, and several emerging technologies could help to achieve this aim.
Authors should adhere to the instructions set by the journals when they prepare their manuscripts. Penelope is an automated tool that enables journals to check manuscripts and gives authors immediate feedback before they submit their work.
Reviewers trained in epidemiology and statistics, of younger age, and who spent up to 3 hours on a review, but not beyond, have been shown to be associated with higher review quality in 1 study.
Using standard checklists by reviewers rather than giving a report in their own words could be helpful. Publons (https://publons.com/home/), a platform focusing on peer review, has developed the Publons Academy, which organizes training courses on peer review practice for young researchers.
The opinion of editors could be influenced by several factors, such as personal beliefs, society's norms, and ethical considerations. Few studies on the behavior of editors have been published, and the little we know in this domain is from what they reveal themselves.
However, a judicious editor can play an adjudicating role and prevent many of the potential abuses of reviewers, such as unsound criticisms, aggressive comments, and deliberate delay of competitors' manuscripts.
Few publishers make the peer review material of their published papers available to evaluate its impact, and there is too little sound research on this approach.
The American Association for the Advancement of Science, the publisher of Science, announced the PRE (Peer Review Evaluation) action, which allows readers to have access to peer review data of published works. Lee and Moher,
specialists in publication science at the University of Washington and Ottawa Hospital Research Institute, respectively, encourage publishers to open what they name “the black box of peer review.” They believe that it is now the ripe time to conduct experimental studies and systematic reviews on various peer review practices.
The International Congress on Peer Review and Scientific Publication, launched by the American Medical Association and the British Medical Association in 1986, encourages research into the quality and credibility of peer review and scientific publication. The activities of this congress are reported in special themed issues of JAMA. The last session of this congress, which was held in September 2017 (http://www.peerreviewcongress.org), encouraged systematic research on peer review and hosted representatives of several journal editors and publishers.
Policy makers should also become interested and invest in peer review. The European Union has launched the project PEERE (new frontiers of peer review, http://www.peere.org). PEERE is an international project involving multiple disciplines and running from 2014 through 2018. The project pools qualitative and quantitative studies on the peer review process and aims to make evidence-based recommendations in this domain.
As long as humans take part in the evaluation of the scientific quality of a manuscript, a completely bias-free system is unlikely. By putting forward sound criticism and suggestions for solutions, these collective efforts could lead to a more accurate and transparent peer review process.
Despite these limitations and concerns, the peer review process remains an invaluable tool to improve research, channel appropriately novel findings and observations, and detect fraud when it occurs.
Dr Arnold Relman, former editor of the New England Journal of Medicine, almost 30 years ago eloquently stated: “Despite the limitations of peer review process, we need it. It is all we have, and it is hard to imagine how we could get along without it.”
More experimental studies and evidence syntheses on biases introduced by the peer review process and its influence on the evidence seen and used by stakeholders are needed, and these could be used to recommend more research in this domain.
Conclusion
The peer review process is far from perfect, and all of its systems (single-blind, double-blind, triple-blind, quadruple-blind, or open) are entwined with advantages and challenges. A double-blind peer review system is supported on equipoise and fair-play principles. Open peer review has a small favorable effect on the quality of the report. There is too little sound research on peer review, and it is time now to perform experimental studies and evidence syntheses of biases introduced by peer review to improve its quality, transparency, and accountability.
Acknowledgments
Drs Haffar and Bazerbachi contributed equally to this work and are co-first authors.
Make per review scientific: thirty years on from the first congress on peer review, Drummond Rennie reflects on the improvements brought about by research into the process--and calls for more.
30707-9&id=gr1.jpg){kind=link}