To The Editor:
The coronavirus disease 2019 (COVID-19) pandemic has sparked an explosion in biological and medical research worldwide, with more than 30,000 COVID- 19 PubMed-indexed publications and 6200 medRxiv and bioRxiv preprints this year as of July 9th, 2020.
1
,National Library of Medicine National Center for Biotechnology Information
NCBI SARS-CoV-2 Resources.
NCBI SARS-CoV-2 Resources.
https://www.ncbi.nlm.nih.gov/sars-cov-2
Date accessed: July 9, 2020
2
However, while immediate access to the latest research is undoubtedly crucial in combating this pandemic, the scientific community must be wary of the potential tradeoff between quantity and quality. In this letter, we discuss the implications of this surge in the rate of publications and suggest measures to minimize undesirable consequences.MedRxiv
COVID-19 SARS-CoV-2 preprints from medRxiv and bioRxiv.
COVID-19 SARS-CoV-2 preprints from medRxiv and bioRxiv.
https://connect.medrxiv.org/relate/content/181
Date accessed: August 31, 2020
During the first week of June 2020, the editors of The Lancet and The New England Journal of Medicine (NEJM), 2 premiere medical journals, both published expressions of concern regarding the questionable quality and reliability of data presented in articles published in their respective journals.
3
,4
Both of the articles in question were published using data from the Surgical Outcomes Collaborative (Surgisphere Corporation, Chicago, IL). In addition, a third article, first uploaded as a preprint in the Social Science Research Network, also used data from Surgisphere. All 3 articles concerned drug therapy in COVID-19 and findings reported in these publications had a major impact on both ongoing clinical patient management and the direction of several clinical trials. The World Health Organization chose to err on the side of caution and suspended the hydroxychloroquine arm of the Solidarity Trial (an ongoing international clinical trial exploring treatments for COVID-19), thus delaying the provision of evidence-based guidelines to practitioners worldwide. The consequences of this occurrence can be exceedingly harmful, and the significant doubts raised regarding these papers has called into question the publishing behavior of both researchers and journals.With the eagerness to publish COVID-19 research, there is a looming potential for errors on the part of both the researchers and the journals. For the researcher, this may involve publishing research of subpar methodological quality or doubtful ethical considerations, or publishing in predatory medical journals.
6
,7
In addition, researchers are rushing to upload their work as preprints, opening the floodgates to a “preprint surge.” While preprints do offer a legitimate platform to share and gain quick feedback on novel results, the haste to preprint articles invites an inherent tradeoff between speed and accuracy.8
,9
The non-peer-reviewed nature of preprints may often promote the spread of inaccurate information, with flawed data often finding itself under the public spotlight through social media and news outlets.8
,9
As the death toll from the pandemic surges, desperation may force clinicians to turn to insufficiently peer-reviewed research for decision-making purposes. However, even peer-reviewed research published by legitimate journals may be called into question. With many highly reputable journals offering fast-tracked services for COVID-19 publications, the robustness of the methodological examination and critical appraisal within the peer review of submissions may be compromised.9
This is evident in the redundancy of COVID-19 research, which in addition to being repetitive, may also be inaccurate, inconsistent, biased, and biologically implausible.10
The torrent of exaggerated or subpar research being published may lead to dangerous consequences and threaten the evidence-based response to COVID-19 globally.With this pandemic leaving health care systems in disarray, healthcare professionals on the frontline face many challenges, ranging from the lack of protective equipment to the overwhelming patient load. It is the collective responsibility of the academic community to make sure that accessing accurate information does not become another challenge. Much of the responsibility for ensuring high quality research lies with the researchers themselves. While one would like to believe that global health and advancing science are the sole motivators to conduct and publish research, some of the COVID-19 research may be due to researchers wanting to jump on the bandwagon for personal gain, such as improving one's resume or attracting media and public attention. This behavior must be countered by a paradigm cultural shift within the academic community, where the quality of research and publishing journals adds more to a researcher’s value than the quantity of publications. Additionally, the academic community must work together towards identifying and denouncing journals that are of a predatory nature and perhaps discount the value of publications in such journals when considering matters such as academic appointments and promotions.
Given the potential harm posed by the publication of articles of subpar quality, reputable medical journals must also consider additional mechanisms to ensure the quality of their fast-tracked peer review process and publication services. These measures could include a preliminary assessment of the submitted work for data sources, data integrity, and research methodology by either editorial staff or external service providers with expertise in this domain. This process may be similar to how news organizations have either internal or external mechanisms for fact-checking news reports or political speech.
In the particular case of Surgisphere, the health care analytics company that provided the data for the two fast-tracked manuscripts published in The Lancet and NEJM,
3
,4
the enticing data set provided by the company included nearly 100,000 detailed patient records from 700 hospitals on 6 continents. This should have raised some red flags as it is questionable how a small analytics firm with few employees, an almost non-existent track record for publications, and no publicly named scientific board, could have obtained reliable patient-level data from hundreds of hospitals on 6 continents in a regulatory environment that strictly limits sharing of protected health information. Moreover, none of the hospitals confirmed working with Surgisphere. The repute of the analytics company and the trustworthiness of the provided data was never questioned because such source checking is not part of the routine peer review process and reviewers are neither trained nor have expertise in this domain. Careful scrutiny of the databases and their sources is even more important as medicine moves towards big data, machine learning, and artificial intelligence. On the other hand, analyses based on data from more reliable and vetted sources such as the National (Nationwide) Inpatient Sample,11
Center for Medicare and Medicaid Services data,Healthcare Cost and Utilization Project
NIS Database Documentation.
NIS Database Documentation.
https://www.hcup-us.ahrq.gov/db/nation/nis/nisdbdocumentation.jsp
Date accessed: July 9, 2020
12
and data from clinical trials registered at ClinicalTrials.govCenter for Medicare and Medicaid Services Database.
https://data.cms.gov
Date accessed: July 9, 2020
13
could be used more reliably for fast-tracked publications in reputable journals.US National Library of Medicine
ClinicalTrials.gov website.
ClinicalTrials.gov website.
https://clinicaltrials.gov
Date accessed: July 9, 2020
The surge in the use of preprint servers during this pandemic also warrants a cautious approach. While preprint servers provide a valuable service and are critical in disseminating timely information to readers during a pandemic such as COVID-19, the non-peer-reviewed nature of the manuscripts posted on these servers, their findings, and the resultant conclusions must be fully acknowledged by the medical community and media. Reputable medical journals should clearly highlight the citing of preprints in the manuscripts submitted to their journal for publication. Similarly, both clinicians and researchers must exercise caution when disseminating unvetted evidence to the public, especially on social media platforms.
The race to publish during this pandemic may cause both the medical community and the public to lose faith in medical research. Therefore, it is imperative that the academic community takes full responsibility for the work that is published, as well as its potential implications, and strives towards ensuring the publication and dissemination of only the highest quality research during these trying times.
Acknowledgments
We acknowledge the Research and Development Wing of the Society for Promoting Innovation in Education (SPIE) at the Center for Innovation in Medical Education at the Aga Khan University for providing mentorship to author Daniyaal Ahmad Cheema on this Letter to the Editor. SPIE is involved with research and innovation in the academic and public health sectors.
References
- NCBI SARS-CoV-2 Resources.https://www.ncbi.nlm.nih.gov/sars-cov-2Date accessed: July 9, 2020
- COVID-19 SARS-CoV-2 preprints from medRxiv and bioRxiv.https://connect.medrxiv.org/relate/content/181Date accessed: August 31, 2020
- Expression of concern: hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis.The Lancet. 2020; 395: e102
- Expression of Concern: Mehra MR et al. Cardiovascular Disease, Drug Therapy, and Mortality in Covid-19.N Engl J Med. 2020;
- Usefulness of Ivermectin in COVID-19 Illness. Available at SSRN 3580524.2020 (382(25):2464)
- COVID-19 coronavirus research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine.J Clin Epidemiol. 2020; 123: 120-126
- Publishing research during pandemics: are you vulnerable to the COVID-19 or predatory publishers.J Pak Med Assoc. 2020; 70: S166-S168
- Coronavirus and the risks of 'speed science.' World Economic Forum website.(Published March 24, 2020. Accessed July 9, 2020)
- Parallel pandemic: the crush of covid-19 publications tests the capacity of scientific publishing. The BMJ Opinion website.(Published May 26, 2020. Accessed July 9, 2020)
- Redundancy in reporting on COVID-19.Eur J Clin Invest. 2020; 50: e13257
- NIS Database Documentation.https://www.hcup-us.ahrq.gov/db/nation/nis/nisdbdocumentation.jspDate accessed: July 9, 2020
- Center for Medicare and Medicaid Services Database.https://data.cms.govDate accessed: July 9, 2020
- ClinicalTrials.gov website.https://clinicaltrials.govDate accessed: July 9, 2020
Article Info
Footnotes
Potential Competing Interests: Dr Sohail reports receiving consulting fees from Medtronic Inc and AziyovBiologics. The other authors report no competing interests.
Identification
Copyright
© 2020 Mayo Foundation for Medical Education and Research
