Advertisement
Mayo Clinic Proceedings Home

Peer Review Bias: A Critical Review

Published:February 20, 2019DOI:https://doi.org/10.1016/j.mayocp.2018.09.004

      Abstract

      Various types of bias and confounding have been described in the biomedical literature that can affect a study before, during, or after the intervention has been delivered. The peer review process can also introduce bias. A compelling ethical and moral rationale necessitates improving the peer review process. A double-blind peer review system is supported on equipoise and fair-play principles. Triple- and quadruple-blind systems have also been described but are not commonly used. The open peer review system introduces “Skin in the Game” heuristic principles for both authors and reviewers and has a small favorable effect on the quality of published reports. In this exposition, we present, on the basis of a comprehensive literature search of PubMed from its inception until October 20, 2017, various possible mechanisms by which the peer review process can distort research results, and we discuss the evidence supporting different strategies that may mitigate this bias. It is time to improve the quality, transparency, and accountability of the peer review system.

      Abbreviations and Acronyms:

      BMJ (The British Medical Journal), JAMA (The Journal of the American Medical Association), OR (odds ratio), RCT (randomized controlled trial), SMD (standardized mean difference)
      Article Highlights
      • Various types of bias and confounding have been described in the biomedical literature.
      • Bias attributable to the peer review process is less recognized.
      • Different peer review systems exist, and some may introduce less bias than others.
      • Experimental studies are needed to identify the least biased peer review approach.
      • We explore mechanisms of peer review bias and offer potential solutions.
      • A compelling ethical and moral rationale necessitates improving the quality, transparency, and accountability of the peer review process.
      The first documented description of a peer review process to regulate the medical profession was in a book titled Ethics of the Physician (Adab al-T˙abīb) written by the Syrian physician Ishāq bin Ali Al-Rahawi (854-931 A.D.) from Al Raha in northern Syria.
      • Al Kawi M.Z.
      History of medical records and peer review.
      This book indicates that one of the duties of a physician is to write duplicate notes for their patients. When the patient had recovered or had died, one of these notes was to be examined by a senior medical council to assess whether malpractice had taken place according to the standards of that time. Peer review to adjudicate scientific publications can be traced back to 1752, when the Royal Society of London created a committee to assess the papers submitted to their journal Philosophical Transactions. The objective was to assist the editor in choosing appropriately themed articles and not to strictly assess their quality.
      • Spier R.
      The history of the peer-review process.
      The scientific peer review process in its actual form began in the United States in the early 1900s, and Science, The Journal of the American Medical Association (JAMA), and American Practitioner were the first journals to implement it. This process was greatly facilitated by the introduction of photocopiers in 1958, which allowed the circulation of several copies of submitted manuscripts to external reviewers without losing the original documents.
      • Castillo M.
      Peer review: past, present, and future.
      Peer review is perceived as contributing to science in 2 ways: assisting the board of editors to accept or reject a paper and helping to improve submitted manuscripts by eliminating major flaws and gaps.
      • van Rooyen S.
      • Black N.
      • Godlee F.
      Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts.
      • Tennant J.P.
      The state of the art in peer review.
      Considering the important effect of the peer review process on the integrity of research, we conducted a narrative critical review of studies that evaluated peer review bias in biomedical research.

      Methods

      We searched PubMed from its inception until October 20, 2017, using the Boolean MeSH terms peer review AND bias. We limited the search to articles written in English having the Boolean MeSH terms in the title and abstract. In addition, we manually searched the reference lists of relevant publications for additional articles.

      Peer Review as a Source of Bias

      Scientific experiments in general, and biomedical research specifically, are subject to bias and confounding. Domains for the different types of bias and confounding have been established as to their role before, during, and after the intervention is delivered.
      • Sterne J.A.
      • Hernán M.A.
      • Reeves B.C.
      • et al.
      ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
      Dedicated tools have been developed to evaluate the bias risk in randomized controlled trials (RCTs), observational studies, diagnostic studies, and others. The role of risk of bias domains has been studied and its effect has been estimated. For example, in randomized trials with inadequate allocation concealment or lack of double-blinding, odds ratios were exaggerated by 41% and 17%, respectively.
      • Schulz K.F.
      • Chalmers I.
      • Hayes R.J.
      • Altman D.G.
      Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials.
      Conceptually, the peer review process can lead to distortion of the results from the viewpoint of the evidence user, akin to bias. Peer review bias can be defined as a violation of impartiality in the evaluation of a submission.
      • Lee C.J.
      • Sugimoto C.R.
      • Zhang G.
      • Cronin B.
      Bias in peer review.
      We propose that this transgression of neutrality standards can affect the dissemination of research (ie, denying publication, which would be a form of publication bias), or modify the spin and communication of findings (affecting the impact and implementation of research), and potentially introduce other unknown biases. This concern was demonstrated in an illustrative study in the journal Brain and Behavior Sciences by Peters and Ceci in 1982.
      • Peters D.
      • Ceci S.
      Peer review practices in psychological journals: the fate of published articles, submitted again.
      Peer review bias is difficult to detect and quantify. One can even argue that such distortion may not fit the classical definition of bias as a unidirectional systematic deviation from truth. On occasions, it may. Peer reviewers can ask authors to delete outcomes, combine outcomes, modify analyses, perform post hoc subgroup analyses, and perform other actions that can, in fact, introduce classical bias. A proposed framework to illustrate a possible role for the peer review process to introduce bias is depicted in the Figure.
      Figure thumbnail gr1
      FigureProposed framework demonstrating how the peer review process can introduce various types of bias.
      The effect of peer reviewers that is mediated by the classical publication bias is intuitive. Nonsignificant results can be denied publication by peer reviewers and may never get published or may go through multiple journals, leading to delay and eventual publication in journals with limited visibility or a low impact factor or in a language other than English.
      • Murad M.H.
      • Montori V.M.
      • Ioannidis J.P.
      • et al.
      How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature.
      Akin to how traditional bias can occur before, during, or after research, peer review bias can be viewed in 3 phases. The “before” phase includes the editor's appraisal of articles before sending them to peer review and making judgments based on the country of origin, institution, or a renowned author. Decisions by the editorial office regarding which peer reviewers to seek can also introduce bias. The “during” phase includes the dogma of reviewers, differences in the proposed paradigm with the authors, and preconceived notions by peer reviewers. The “after” phase includes stalling and delaying a publication or perhaps writing an editorial that might mitigate/negate/temper the findings of the study.
      The effect of authors' conflicts of interest on studies' evaluations is well known and has been reported by Chaudhry et al.
      • Chaudhry S.
      • Schroter S.
      • Smith R.
      • Morris J.
      Does declaration of competing interests affect readers' perceptions? a randomised trial.
      The significance of reviewers' conflicts is increasingly recognized, and most editors now ask their reviewers to report their conflicts of interest before reviewing or to contact the editors in case of doubt.
      • Bourne P.E.
      • Korngreen A.
      Ten simple rules for reviewers.
      Many other less-known mechanisms by which peer review can introduce bias have been described.
      • Lee C.J.
      • Sugimoto C.R.
      • Zhang G.
      • Cronin B.
      Bias in peer review.

      Systems of Scientific Peer Review

      There are several systems of scientific peer review: single-blind, double-blind, triple-blind, quadruple-blind, and open review.
      • O'Connor E.
      • Cousar M.
      • Lentini J.
      • Castillo M.
      • Halm K.
      • Zeffiro T.
      Efficacy of double-blind peer review in an imaging subspecialty journal.
      • Groves T.
      • Loder E.
      Prepublication histories and open peer review at The BMJ.

       Single-Blind Peer Review

      In single-blind peer review, the reviewers know the identity of the authors but not vice versa. Single-blind peer review has remained the norm in clinical medicine and biomedical journals.
      • Chung K.C.
      • Shauver M.J.
      • Malay S.
      • Zhong L.
      • Weinstein A.
      • Rohrich R.J.
      Is double-blinded peer review necessary? the effect of blinding on review quality.
      The primary advantage of single-blind peer review is that reviewers are free from authors' influence, which enables them to be straightforward in their comments and appraisals without fear of conflicts.
      • O'Connor E.
      • Cousar M.
      • Lentini J.
      • Castillo M.
      • Halm K.
      • Zeffiro T.
      Efficacy of double-blind peer review in an imaging subspecialty journal.
      Although knowledge of authors' identities and their institutions is advantageous to famous authors and authors from prestigious institutions, it is also detrimental to junior investigators.
      • Tomkins A.
      • Zhang M.
      • Heavlin W.D.
      Reviewer bias in single- versus double-blind peer review.
      Discrimination against authors because of their nationality, native language, sex, or institution is plausible.
      • Rowland F.
      The peer-review process.

       Double-Blind Peer Review

      In double-blind peer review, the identities of both authors and reviewers are kept hidden from each other. Double-blind peer review has been used in imaging, nursing, and humanities journals.
      • Chung K.C.
      • Shauver M.J.
      • Malay S.
      • Zhong L.
      • Weinstein A.
      • Rohrich R.J.
      Is double-blinded peer review necessary? the effect of blinding on review quality.
      Double-blind peer review is considered more objective and fair, while still offering all the advantages of single-blind peer review. Although it is plausible that double-blind peer review would protect the authors from reviewers' bias, this premise requires substantiation with stronger evidence. Certain aspects of reviewers' abuse persist whether in a single- or double-blind peer review system, such as plagiarism of nonpublished data for personal benefit or deliberate delay of publication to usurp the authors.
      • Rowland F.
      The peer-review process.
      A reviewer may take a longer time in a double-blind system because he or she may spend a lot of time uncovering the author's identity instead of focusing on proper review of the manuscript.
      • Chung K.C.
      • Shauver M.J.
      • Malay S.
      • Zhong L.
      • Weinstein A.
      • Rohrich R.J.
      Is double-blinded peer review necessary? the effect of blinding on review quality.
      True blinding could be difficult to accomplish. Reviewers can identify the authors successfully in 25% to 50% of cases in biomedical or social science journals.
      • O'Connor E.
      • Cousar M.
      • Lentini J.
      • Castillo M.
      • Halm K.
      • Zeffiro T.
      Efficacy of double-blind peer review in an imaging subspecialty journal.
      Common reasons for blinding failure are reviewer familiarity with authors' work and when the authors inadvertently reveal their identity in the manuscript.
      • Krinsky G.
      How to avoid “unblinding” the peer review process.
      Phrases such as “we have shown that” should be replaced by “previous studies have shown that.” Authors are advised to avoid the use of self-citations and citations that are “in press” because such practices indicate that the work is done by the same authors. Furthermore, authors should remove their initials from the manuscript and remove names of institutions from images and drawings.
      • Krinsky G.
      How to avoid “unblinding” the peer review process.
      Although it could be argued that the double-blind peer review process adds burden to the editorial staff workload and may increase publishing cost, in the era of electronic submission methods and publication checklists, some of these theoretical concerns can be addressed.

       Triple-Blind Peer Review

      In this model, in addition to blinding the reviewers' identity, the authors' identity is also hidden from the editors during the submission process, and a deidentification code is assigned. Thereafter, the editorial board guides the manuscript without knowledge of the authors' names or institutions. All communications occur through the journal's website without divulging any author-related identifying features. This approach is implemented by numerous academic journals in the fields of philosophy (eg, Philosophy and Phenomenological Research, The British Journal for the Philosophy of Science) and quality improvement (eg, BMJ Quality & Safety) and has reportedly been trialed recently by the journal Molecular Cell.
      • da Silva J.A.T.
      • Dobránszki J.
      The role of the anonymous voice in post-publication peer review versus traditional peer review.

       Quadruple-Blind Peer Review

      The steps of double-blind and triple-blind peer review processes are preserved and augmented by hiding the identity of the handling editor. The anonymous editor is appointed by the editorial board committee, and his or her identity remains confidential throughout. One attempt to reflect a quadruple-blind peer review process has been demonstrated in the submission policy of Ethics.
      • Teixeira da Silva J.A.
      • Al-Khatib A.
      • Katavić V.
      • Bornemann-Cimenti H.
      Establishing sensible and practical guidelines for desk rejections.

       Open Peer Review

      In an open peer review system, both authors and reviewers know the identity of each other. Whereas The British Medical Journal (BMJ) succeeded in moving to an open peer review system nearly 2 decades ago, Nature and PLOS Medicine have had less successful attempts because of nonengagement of the authors and reviewers.
      • Lee C.J.
      • Sugimoto C.R.
      • Zhang G.
      • Cronin B.
      Bias in peer review.
      Supporters of this system believe that it increases transparency, encourages reviewers to be constructive in their comments, and could prevent plagiarism.
      • O'Connor E.
      • Cousar M.
      • Lentini J.
      • Castillo M.
      • Halm K.
      • Zeffiro T.
      Efficacy of double-blind peer review in an imaging subspecialty journal.
      This system introduces the “Skin in the Game” principle for both authors and reviewers.
      • Taleb N.N.
      • Sandis C.
      The skin in the game heuristic for protection against tail events.
      The main disadvantage of open review is the possible creation of animosities and retaliation between authors and reviewers.
      • DeCoursey T.
      The pros and cons of open peer review: should authors be told who their reviewers are?.
      The open peer review system is associated with a higher refusal rate from reviewers and an increased time to write the reviews. Some journals ask the authors to suggest reviewers, but these reviewers may be more favorable than those nominated by the editors. Furthermore, this approach may be conducive to scientific fraud.
      • Haug C.J.
      Peer-review fraud: hacking the scientific publication process.
      Many journals are implementing the open peer review system despite the lack of supporting evidence. Seventy journals from BioMed Central currently adopt different models of open peer review.
      • Papatriantafyllou M.
      Peer review: the future is here.

      Which System of Scientific Peer Review Is Better?

      It is difficult to answer this question with certitude. Each system (single-blind, double-blind, triple-blind, quadruple-blind, or open) has advantages and challenges.
      • Bauchner H.
      • Fontanarosa P.
      Bias among peer reviewers.
      However, double-blind and open peer reviews are more supported at present.
      • Tennant J.P.
      The state of the art in peer review.
      • Lee C.J.
      • Sugimoto C.R.
      • Zhang G.
      • Cronin B.
      Bias in peer review.
      • Tomkins A.
      • Zhang M.
      • Heavlin W.D.
      Reviewer bias in single- versus double-blind peer review.
      • Bekkering G.E.
      • Kleijnen J.
      Procedures and methods of benefit assessments for medicines in Germany.
      Furthermore, the double-blind peer review system seems less biased at face value. Burdens on editorial staff can be reduced by a checklist system and software developments (eg, having separate entries for authors' names, institutions, and the body of the manuscript that yields automatically an unidentified manuscript for circulation).
      A systematic review and meta-analysis of RCTs that evaluated the impact of different interventions on the quality of peer review has been published recently.
      • Bruce R.
      • Chauvin A.
      • Trinquart L.
      • Ravaud P.
      • Boutron I.
      Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis.
      There was no statistically significant difference in the quality of peer review report and recommendations for rejection between single- and double-blind systems. However, the absence of a statistically significant difference should not be interpreted as a lack of blinding benefit because this body of evidence warrants low certainty (a high level of heterogeneity, a small number of RCTs, and insufficient data to conduct important subgroup analyses). Open peer review had a small favorable effect on the quality of peer review report and decreased recommendations for rejection (Table).
      TableBlind vs Open Peer Review Systems
      OR = odds ratio; RCT = randomized controlled trial; SMD = standardized mean difference.
      Summarized from a meta-analysis by Bruce et al.27
      Assessed interventionNo. of RCTsNo. of manuscripts in each armSMD or OR (95% CI)I2 for heterogeneity (%)
      Single- vs double-blind peer review systems
       Quality of peer review report3513/511SMD: 0.12 (−0.12 to 0.36)68
       Recommendations for rejection3282/282OR: 0.77 (0.39 to 1.50)69
      Double-blind vs open peer review systems
       Quality of peer review report3489/459SMD: 0.13 (0.00 to 0.26)0
       Recommendations for rejection4582/514OR: 0.47 (0.28 to 0.82)63
      a OR = odds ratio; RCT = randomized controlled trial; SMD = standardized mean difference.
      b Summarized from a meta-analysis by Bruce et al.
      • Bruce R.
      • Chauvin A.
      • Trinquart L.
      • Ravaud P.
      • Boutron I.
      Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis.
      Revelations of fabricated data in the New England Journal of Medicine and other prestigious journals have shown that peer review was unable to detect fraud.
      • Culliton B.J.
      Coping with fraud: the Darsee Case.
      More recent work illustrates how easily fraudulent data continue to be published.
      • Rennie D.
      Make per review scientific: thirty years on from the first congress on peer review, Drummond Rennie reflects on the improvements brought about by research into the process--and calls for more.
      The latter example of scientific misconduct took place over several years and affected at least 30 publications in peer-reviewed journals. Fraud in most cases is the work of individuals, not a team, and it is thought that close supervision by senior scientists participating in clinical trials could preclude such regrettable events.
      • Relman A.S.
      Peer review in scientific journals: what good is it?.
      The peer review process is far from perfect. Dr Richard Smith, former editor of BMJ, described the single-blind peer review system in an editorial published in 1999 as “slow, expensive, highly subjective, prone to bias, easily abused, and poor at detecting gross defects.”
      • Smith R.
      Opening up BMJ peer review: a beginning that should lead to complete transparency.

      How to Improve the Peer Review Process?

      The input from a peer reviewer is essential to ensure that articles are valid and accurately reported,
      • Bauchner H.
      • Fontanarosa P.
      Bias among peer reviewers.
      and the most important question in this regard is not to abandon it but how to improve it.
      Peer review involves several groups: authors, who want to publish their work for career promotion or to procure funding; editors, who are on the lookout for novel and sound research; reviewers, who try to find sufficient time in their busy schedules and often without remuneration or academic credit; and publishers, who compete with other rivals in a fast-changing field.
      • Papatriantafyllou M.
      Peer review: the future is here.
      The ultimate stakeholder in this process is the patient, whose care and even survival may depend on the sound flow of the peer review process. These groups have to be engaged collectively to improve the peer review process, and several emerging technologies could help to achieve this aim.
      Authors should adhere to the instructions set by the journals when they prepare their manuscripts. Penelope is an automated tool that enables journals to check manuscripts and gives authors immediate feedback before they submit their work.
      • Harwood J.
      Start-up story: Penelope.
      It would be reasonable to give authors the opportunity to appeal against the decision of editors and reviewers when they find it to be unfair.
      • Newton D.P.
      Quality and peer review of research: an adjudicating role for editors.
      Reviewers trained in epidemiology and statistics, of younger age, and who spent up to 3 hours on a review, but not beyond, have been shown to be associated with higher review quality in 1 study.
      • Black N.
      • Van Rooyen S.
      • Godlee F.
      • Smith R.
      • Evans S.
      What makes a good reviewer and a good review for a general medical journal?.
      Using standard checklists by reviewers rather than giving a report in their own words could be helpful. Publons (https://publons.com/home/), a platform focusing on peer review, has developed the Publons Academy, which organizes training courses on peer review practice for young researchers.
      The opinion of editors could be influenced by several factors, such as personal beliefs, society's norms, and ethical considerations. Few studies on the behavior of editors have been published, and the little we know in this domain is from what they reveal themselves.
      • Newton D.P.
      Quality and peer review of research: an adjudicating role for editors.
      However, a judicious editor can play an adjudicating role and prevent many of the potential abuses of reviewers, such as unsound criticisms, aggressive comments, and deliberate delay of competitors' manuscripts.
      • O'Connor E.
      • Cousar M.
      • Lentini J.
      • Castillo M.
      • Halm K.
      • Zeffiro T.
      Efficacy of double-blind peer review in an imaging subspecialty journal.
      A code of practice for editors is advocated by some and could be helpful.
      • Newton D.P.
      Quality and peer review of research: an adjudicating role for editors.
      Few publishers make the peer review material of their published papers available to evaluate its impact, and there is too little sound research on this approach.
      • Lee C.J.
      • Moher D.
      Promote scientific integrity via journal peer review data.
      The American Association for the Advancement of Science, the publisher of Science, announced the PRE (Peer Review Evaluation) action, which allows readers to have access to peer review data of published works. Lee and Moher,
      • Lee C.J.
      • Moher D.
      Promote scientific integrity via journal peer review data.
      specialists in publication science at the University of Washington and Ottawa Hospital Research Institute, respectively, encourage publishers to open what they name “the black box of peer review.” They believe that it is now the ripe time to conduct experimental studies and systematic reviews on various peer review practices.
      The International Congress on Peer Review and Scientific Publication, launched by the American Medical Association and the British Medical Association in 1986, encourages research into the quality and credibility of peer review and scientific publication. The activities of this congress are reported in special themed issues of JAMA. The last session of this congress, which was held in September 2017 (http://www.peerreviewcongress.org), encouraged systematic research on peer review and hosted representatives of several journal editors and publishers.
      Policy makers should also become interested and invest in peer review. The European Union has launched the project PEERE (new frontiers of peer review, http://www.peere.org). PEERE is an international project involving multiple disciplines and running from 2014 through 2018. The project pools qualitative and quantitative studies on the peer review process and aims to make evidence-based recommendations in this domain.
      As long as humans take part in the evaluation of the scientific quality of a manuscript, a completely bias-free system is unlikely. By putting forward sound criticism and suggestions for solutions, these collective efforts could lead to a more accurate and transparent peer review process.
      • Papatriantafyllou M.
      Peer review: the future is here.
      Despite these limitations and concerns, the peer review process remains an invaluable tool to improve research, channel appropriately novel findings and observations, and detect fraud when it occurs.
      • Smith R.
      Peer review: a flawed process at the heart of science and journals.
      Dr Arnold Relman, former editor of the New England Journal of Medicine, almost 30 years ago eloquently stated: “Despite the limitations of peer review process, we need it. It is all we have, and it is hard to imagine how we could get along without it.”
      • Relman A.S.
      Peer review in scientific journals: what good is it?.
      More experimental studies and evidence syntheses on biases introduced by the peer review process and its influence on the evidence seen and used by stakeholders are needed, and these could be used to recommend more research in this domain.

      Conclusion

      The peer review process is far from perfect, and all of its systems (single-blind, double-blind, triple-blind, quadruple-blind, or open) are entwined with advantages and challenges. A double-blind peer review system is supported on equipoise and fair-play principles. Open peer review has a small favorable effect on the quality of the report. There is too little sound research on peer review, and it is time now to perform experimental studies and evidence syntheses of biases introduced by peer review to improve its quality, transparency, and accountability.

      Acknowledgments

      Drs Haffar and Bazerbachi contributed equally to this work and are co-first authors.

      Supplemental Online Material

      References

        • Al Kawi M.Z.
        History of medical records and peer review.
        Ann Saudi Med. 1997; 17: 277-278
        • Spier R.
        The history of the peer-review process.
        Trends Biotechnol. 2002; 20: 357-358
        • Castillo M.
        Peer review: past, present, and future.
        AJNR Am J Neuroradiol. 2012; 33: 1833-1835
        • van Rooyen S.
        • Black N.
        • Godlee F.
        Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts.
        J Clin Epidemiol. 1999; 52: 625-629
        • Tennant J.P.
        The state of the art in peer review.
        FEMS Microbiol Lett. 2018; 365
        • Sterne J.A.
        • Hernán M.A.
        • Reeves B.C.
        • et al.
        ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
        BMJ. 2016; 355: i4919
        • Schulz K.F.
        • Chalmers I.
        • Hayes R.J.
        • Altman D.G.
        Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials.
        JAMA. 1995; 273: 408-412
        • Lee C.J.
        • Sugimoto C.R.
        • Zhang G.
        • Cronin B.
        Bias in peer review.
        J Am Soc Inf Sci Tec. 2013; 64: 2-17
        • Peters D.
        • Ceci S.
        Peer review practices in psychological journals: the fate of published articles, submitted again.
        Behav Brain Sci. 1982; 5: 187-255
        • Murad M.H.
        • Montori V.M.
        • Ioannidis J.P.
        • et al.
        How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature.
        JAMA. 2014; 312: 171-179
        • Chaudhry S.
        • Schroter S.
        • Smith R.
        • Morris J.
        Does declaration of competing interests affect readers' perceptions? a randomised trial.
        BMJ. 2002; 325: 1391-1392
        • Bourne P.E.
        • Korngreen A.
        Ten simple rules for reviewers.
        PLoS Comput Biol. 2006; 2: e110
        • O'Connor E.
        • Cousar M.
        • Lentini J.
        • Castillo M.
        • Halm K.
        • Zeffiro T.
        Efficacy of double-blind peer review in an imaging subspecialty journal.
        AJNR Am J Neuroradiol. 2017; 38: 230-235
        • Groves T.
        • Loder E.
        Prepublication histories and open peer review at The BMJ.
        BMJ. 2014; 349: g5394
        • Chung K.C.
        • Shauver M.J.
        • Malay S.
        • Zhong L.
        • Weinstein A.
        • Rohrich R.J.
        Is double-blinded peer review necessary? the effect of blinding on review quality.
        Plast Reconstr Surg. 2015; 136: 1369-1377
        • Tomkins A.
        • Zhang M.
        • Heavlin W.D.
        Reviewer bias in single- versus double-blind peer review.
        Proc Natl Acad Sci U S A. 2017; 114: 12708-12713
        • Rowland F.
        The peer-review process.
        Learned Publishing. 2002; 15: 247-258
        • Krinsky G.
        How to avoid “unblinding” the peer review process.
        AJR Am J Roentgenol. 1999; 172: 1474
        • da Silva J.A.T.
        • Dobránszki J.
        The role of the anonymous voice in post-publication peer review versus traditional peer review.
        KOME: An International Journal of Pure Communication Inquiry. 2015; 3: 90-94
        • Teixeira da Silva J.A.
        • Al-Khatib A.
        • Katavić V.
        • Bornemann-Cimenti H.
        Establishing sensible and practical guidelines for desk rejections.
        Sci Eng Ethics. 2018; 24: 1347-1365
        • Taleb N.N.
        • Sandis C.
        The skin in the game heuristic for protection against tail events.
        Rev Behav Econ. 2014; 1: 115-135
        • DeCoursey T.
        The pros and cons of open peer review: should authors be told who their reviewers are?.
        Nature. 2006; https://doi.org/10.1038/nature04991
        • Haug C.J.
        Peer-review fraud: hacking the scientific publication process.
        N Engl J Med. 2015; 373: 2393-2395
        • Papatriantafyllou M.
        Peer review: the future is here.
        FEBS Lett. 2017; 591: 2789-2792
        • Bauchner H.
        • Fontanarosa P.
        Bias among peer reviewers.
        JAMA. 2017; 318: 755-756
        • Bekkering G.E.
        • Kleijnen J.
        Procedures and methods of benefit assessments for medicines in Germany.
        Eur J Health Econ. 2008; 9: 5-29
        • Bruce R.
        • Chauvin A.
        • Trinquart L.
        • Ravaud P.
        • Boutron I.
        Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis.
        BMC Med. 2016; 14: 85
        • Culliton B.J.
        Coping with fraud: the Darsee Case.
        Science. 1983; 220: 31-35
        • Rennie D.
        Make per review scientific: thirty years on from the first congress on peer review, Drummond Rennie reflects on the improvements brought about by research into the process--and calls for more.
        Nature. 2016; 535: 31-33
        • Relman A.S.
        Peer review in scientific journals: what good is it?.
        West J Med. 1990; 153: 520-522
        • Smith R.
        Opening up BMJ peer review: a beginning that should lead to complete transparency.
        BMJ. 1999; 318: 4
        • Harwood J.
        Start-up story: Penelope.
        Insights. 2017; 30: 96-99
        • Newton D.P.
        Quality and peer review of research: an adjudicating role for editors.
        Account Res. 2010; 17: 130-145
        • Black N.
        • Van Rooyen S.
        • Godlee F.
        • Smith R.
        • Evans S.
        What makes a good reviewer and a good review for a general medical journal?.
        JAMA. 1998; 280: 231-233
        • Lee C.J.
        • Moher D.
        Promote scientific integrity via journal peer review data.
        Science. 2017; 357: 256-257
        • Smith R.
        Peer review: a flawed process at the heart of science and journals.
        J R Soc Med. 2006; 99: 178-182