Advertisement
Mayo Clinic Proceedings Home

The Pandemic of Publications: Are We Sacrificing Quality for Quantity?

      To The Editor:
      The coronavirus disease 2019 (COVID-19) pandemic has sparked an explosion in biological and medical research worldwide, with more than 30,000 COVID- 19 PubMed-indexed publications and 6200 medRxiv and bioRxiv preprints this year as of July 9th, 2020.
      National Library of Medicine National Center for Biotechnology Information
      NCBI SARS-CoV-2 Resources.
      ,
      MedRxiv
      COVID-19 SARS-CoV-2 preprints from medRxiv and bioRxiv.
      However, while immediate access to the latest research is undoubtedly crucial in combating this pandemic, the scientific community must be wary of the potential tradeoff between quantity and quality. In this letter, we discuss the implications of this surge in the rate of publications and suggest measures to minimize undesirable consequences.
      During the first week of June 2020, the editors of The Lancet and The New England Journal of Medicine (NEJM), 2 premiere medical journals, both published expressions of concern regarding the questionable quality and reliability of data presented in articles published in their respective journals.
      The Lancet E
      Expression of concern: hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis.
      ,
      • Rubin E.J.
      Expression of Concern: Mehra MR et al. Cardiovascular Disease, Drug Therapy, and Mortality in Covid-19.
      Both of the articles in question were published using data from the Surgical Outcomes Collaborative (Surgisphere Corporation, Chicago, IL). In addition, a third article, first uploaded as a preprint in the Social Science Research Network, also used data from Surgisphere.
      • Patel A.
      Usefulness of Ivermectin in COVID-19 Illness. Available at SSRN 3580524.
      All 3 articles concerned drug therapy in COVID-19 and findings reported in these publications had a major impact on both ongoing clinical patient management and the direction of several clinical trials. The World Health Organization chose to err on the side of caution and suspended the hydroxychloroquine arm of the Solidarity Trial (an ongoing international clinical trial exploring treatments for COVID-19), thus delaying the provision of evidence-based guidelines to practitioners worldwide. The consequences of this occurrence can be exceedingly harmful, and the significant doubts raised regarding these papers has called into question the publishing behavior of both researchers and journals.
      With the eagerness to publish COVID-19 research, there is a looming potential for errors on the part of both the researchers and the journals. For the researcher, this may involve publishing research of subpar methodological quality or doubtful ethical considerations, or publishing in predatory medical journals.
      • Alexander P.E.
      • Debono V.B.
      • Mammen M.J.
      • Iorio A.
      • Aryal K.
      • Deng D.
      • et al.
      COVID-19 coronavirus research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine.
      ,
      • Memon A.R.
      • Rathore F.A.
      Publishing research during pandemics: are you vulnerable to the COVID-19 or predatory publishers.
      In addition, researchers are rushing to upload their work as preprints, opening the floodgates to a “preprint surge.” While preprints do offer a legitimate platform to share and gain quick feedback on novel results, the haste to preprint articles invites an inherent tradeoff between speed and accuracy.
      • Sharma M.
      • Scarr S.
      • Kelland K.
      Coronavirus and the risks of 'speed science.' World Economic Forum website.
      ,
      • Kurth T.
      • Piccininni M.
      • Loder E.
      • Rohmann J.
      Parallel pandemic: the crush of covid-19 publications tests the capacity of scientific publishing. The BMJ Opinion website.
      The non-peer-reviewed nature of preprints may often promote the spread of inaccurate information, with flawed data often finding itself under the public spotlight through social media and news outlets.
      • Sharma M.
      • Scarr S.
      • Kelland K.
      Coronavirus and the risks of 'speed science.' World Economic Forum website.
      ,
      • Kurth T.
      • Piccininni M.
      • Loder E.
      • Rohmann J.
      Parallel pandemic: the crush of covid-19 publications tests the capacity of scientific publishing. The BMJ Opinion website.
      As the death toll from the pandemic surges, desperation may force clinicians to turn to insufficiently peer-reviewed research for decision-making purposes. However, even peer-reviewed research published by legitimate journals may be called into question. With many highly reputable journals offering fast-tracked services for COVID-19 publications, the robustness of the methodological examination and critical appraisal within the peer review of submissions may be compromised.
      • Kurth T.
      • Piccininni M.
      • Loder E.
      • Rohmann J.
      Parallel pandemic: the crush of covid-19 publications tests the capacity of scientific publishing. The BMJ Opinion website.
      This is evident in the redundancy of COVID-19 research, which in addition to being repetitive, may also be inaccurate, inconsistent, biased, and biologically implausible.
      • Papes D.
      • Ozimec E.
      Redundancy in reporting on COVID-19.
      The torrent of exaggerated or subpar research being published may lead to dangerous consequences and threaten the evidence-based response to COVID-19 globally.
      With this pandemic leaving health care systems in disarray, healthcare professionals on the frontline face many challenges, ranging from the lack of protective equipment to the overwhelming patient load. It is the collective responsibility of the academic community to make sure that accessing accurate information does not become another challenge. Much of the responsibility for ensuring high quality research lies with the researchers themselves. While one would like to believe that global health and advancing science are the sole motivators to conduct and publish research, some of the COVID-19 research may be due to researchers wanting to jump on the bandwagon for personal gain, such as improving one's resume or attracting media and public attention. This behavior must be countered by a paradigm cultural shift within the academic community, where the quality of research and publishing journals adds more to a researcher’s value than the quantity of publications. Additionally, the academic community must work together towards identifying and denouncing journals that are of a predatory nature and perhaps discount the value of publications in such journals when considering matters such as academic appointments and promotions.
      Given the potential harm posed by the publication of articles of subpar quality, reputable medical journals must also consider additional mechanisms to ensure the quality of their fast-tracked peer review process and publication services. These measures could include a preliminary assessment of the submitted work for data sources, data integrity, and research methodology by either editorial staff or external service providers with expertise in this domain. This process may be similar to how news organizations have either internal or external mechanisms for fact-checking news reports or political speech.
      In the particular case of Surgisphere, the health care analytics company that provided the data for the two fast-tracked manuscripts published in The Lancet and NEJM,
      The Lancet E
      Expression of concern: hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis.
      ,
      • Rubin E.J.
      Expression of Concern: Mehra MR et al. Cardiovascular Disease, Drug Therapy, and Mortality in Covid-19.
      the enticing data set provided by the company included nearly 100,000 detailed patient records from 700 hospitals on 6 continents. This should have raised some red flags as it is questionable how a small analytics firm with few employees, an almost non-existent track record for publications, and no publicly named scientific board, could have obtained reliable patient-level data from hundreds of hospitals on 6 continents in a regulatory environment that strictly limits sharing of protected health information. Moreover, none of the hospitals confirmed working with Surgisphere. The repute of the analytics company and the trustworthiness of the provided data was never questioned because such source checking is not part of the routine peer review process and reviewers are neither trained nor have expertise in this domain. Careful scrutiny of the databases and their sources is even more important as medicine moves towards big data, machine learning, and artificial intelligence. On the other hand, analyses based on data from more reliable and vetted sources such as the National (Nationwide) Inpatient Sample,
      Healthcare Cost and Utilization Project
      NIS Database Documentation.
      Center for Medicare and Medicaid Services data,
      Center for Medicare and Medicaid Services Database.
      and data from clinical trials registered at ClinicalTrials.gov
      US National Library of Medicine
      ClinicalTrials.gov website.
      could be used more reliably for fast-tracked publications in reputable journals.
      The surge in the use of preprint servers during this pandemic also warrants a cautious approach. While preprint servers provide a valuable service and are critical in disseminating timely information to readers during a pandemic such as COVID-19, the non-peer-reviewed nature of the manuscripts posted on these servers, their findings, and the resultant conclusions must be fully acknowledged by the medical community and media. Reputable medical journals should clearly highlight the citing of preprints in the manuscripts submitted to their journal for publication. Similarly, both clinicians and researchers must exercise caution when disseminating unvetted evidence to the public, especially on social media platforms.
      The race to publish during this pandemic may cause both the medical community and the public to lose faith in medical research. Therefore, it is imperative that the academic community takes full responsibility for the work that is published, as well as its potential implications, and strives towards ensuring the publication and dissemination of only the highest quality research during these trying times.

      Acknowledgments

      We acknowledge the Research and Development Wing of the Society for Promoting Innovation in Education (SPIE) at the Center for Innovation in Medical Education at the Aga Khan University for providing mentorship to author Daniyaal Ahmad Cheema on this Letter to the Editor. SPIE is involved with research and innovation in the academic and public health sectors.

      References

        • National Library of Medicine National Center for Biotechnology Information
        NCBI SARS-CoV-2 Resources.
        https://www.ncbi.nlm.nih.gov/sars-cov-2
        Date accessed: July 9, 2020
        • MedRxiv
        COVID-19 SARS-CoV-2 preprints from medRxiv and bioRxiv.
        • The Lancet E
        Expression of concern: hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis.
        The Lancet. 2020; 395: e102
        • Rubin E.J.
        Expression of Concern: Mehra MR et al. Cardiovascular Disease, Drug Therapy, and Mortality in Covid-19.
        N Engl J Med. 2020;
        • Patel A.
        Usefulness of Ivermectin in COVID-19 Illness. Available at SSRN 3580524.
        2020 (382(25):2464)
        • Alexander P.E.
        • Debono V.B.
        • Mammen M.J.
        • Iorio A.
        • Aryal K.
        • Deng D.
        • et al.
        COVID-19 coronavirus research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine.
        J Clin Epidemiol. 2020; 123: 120-126
        • Memon A.R.
        • Rathore F.A.
        Publishing research during pandemics: are you vulnerable to the COVID-19 or predatory publishers.
        J Pak Med Assoc. 2020; 70: S166-S168
        • Sharma M.
        • Scarr S.
        • Kelland K.
        Coronavirus and the risks of 'speed science.' World Economic Forum website.
        (Published March 24, 2020. Accessed July 9, 2020)
        • Kurth T.
        • Piccininni M.
        • Loder E.
        • Rohmann J.
        Parallel pandemic: the crush of covid-19 publications tests the capacity of scientific publishing. The BMJ Opinion website.
        (Published May 26, 2020. Accessed July 9, 2020)
        • Papes D.
        • Ozimec E.
        Redundancy in reporting on COVID-19.
        Eur J Clin Invest. 2020; 50: e13257
        • Healthcare Cost and Utilization Project
        NIS Database Documentation.
      1. Center for Medicare and Medicaid Services Database.
        https://data.cms.gov
        Date accessed: July 9, 2020
        • US National Library of Medicine
        ClinicalTrials.gov website.
        https://clinicaltrials.gov
        Date accessed: July 9, 2020

      Linked Article

      • Erratum
        Mayo Clinic ProceedingsVol. 96Issue 4
        • Preview
          Correction to ‘The Pandemic of Publications: Are We Sacrificing Quality for Quantity?’ [Mayo Clinic Proceedings 95 (2020) 2288–2290/3037]
        • Full-Text
        • PDF