Advertisement
Mayo Clinic Proceedings Home

A Framework for Evidence Synthesis Programs to Respond to a Pandemic

      Abstract

      The coronavirus disease 2019 (COVID-19) pandemic requires making rapid decisions based on sparse and rapidly changing evidence. Evidence synthesis programs conduct systematic reviews for guideline developers, health systems clinicians, and decision-makers that usually take an average 6 to 8 months to complete. We present a framework for evidence synthesis programs to respond to pandemics that has proven feasible and practical during the COVID-19 response in a large multistate health system employing more than 78,000 people. The framework includes four components: an approach for conducting rapid reviews, a repository of rapid reviews, a registry for all original studies about COVID-19, and twice-weekly prioritized update of new evidence sent to key stakeholders. As COVID-19 will not be our last pandemic, we share the details of this framework to allow replication in other institutions and re-implementation in future pandemics.
      The coronavirus disease 2019 (COVID-19) pandemic has challenged all parts of the health care enterprise with the need to make rapid decisions based on sparse and rapidly changing evidence.
      • Shah A.
      • Kashyap R.
      • Tosh P.
      • Sampathkumar P.
      • O'Horo J.C.
      Guide to understanding the 2019 novel coronavirus.
      Evidence synthesis programs are research units that conduct systematic reviews for guideline developers, health systems clinicians, and decision-makers; these programs had to significantly change their modus operandi or otherwise become irrelevant to decision-making. A typical systematic review takes an average 6 to 18 months to complete,
      • Smith V.
      • Devane D.
      • Begley C.M.
      • Clarke M.
      Methodology in conducting a systematic review of systematic reviews of healthcare interventions.
      making this approach outdated and impractical during pandemics. Much has been written about rapid reviews as an alternative that trades off some aspects of rigor with being rapid and concise. Surveys of health systems stakeholders have always shown that they preferred rapid reviews.
      • Schoelles K.
      • Umscheid C.A.
      • Lin J.S.
      • et al.
      However, there is no consensus on how to define or conduct rapid reviews,
      • Haby M.M.
      • Chapman E.
      • Clark R.
      • Barreto J.
      • Reveiz L.
      • Lavis J.N.
      What are the best methodologies for rapid reviews of the research evidence for evidence-informed decision making in health policy and practice: a rapid review.
      and little is known about how to make such reviews operational during time of crisis.
      In this exposition, we present a framework for an evidence synthesis program response to a pandemic. This framework has proven feasible and practical during the COVID-19 response in a large multistate academic center employing more than 78,000 people and caring for greater than 1,200,000 patients per year from all 50 US states and 138 countries. As COVID-19 will not be the last pandemic we face, we share this experience to allow replication in other institutions and re-implementation in future pandemics.

      Setting

      Mayo Clinic is a large academic health system with three major campuses located in Minnesota, Florida, and Arizona, as well as multiple Mayo Clinic Health System hospitals and clinics throughout the Midwest in Minnesota and Wisconsin. The evidence synthesis program is one of the Evidence-based Practice Centers designated by the Agency of Healthcare Quality and Research. The program consists of two faculties, librarians, 10 to 15 core investigators, and numerous clinicians and ad hoc collaborators, and is housed within the Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery.

      Response Components

      The evidence synthesis program developed a response within 2 weeks of the World Health Organization declaration of a pandemic due to severe acute respiratory system coronavirus 2 on March 11, 2020. The development of the response followed a quality improvement methodology in which the current state of evidence synthesis was evaluated and found insufficient, relevant stakeholders were identified, and rapid and iterative cycles of intervention were initiated and modified repeatedly based on stakeholders’ feedback. The response consisted of four components (Figure).
      Figure thumbnail gr1
      FigureFramework for evidence synthesis programs to respond to pandemics. The framework includes four components and presents a description of the component and relevant implementation information.

       What is New?

      We called this component of the framework “What is New?” It consists of a list of studies that have been published in the previous 3 days about COVID-19. A search query was developed early in the pandemic via collaboration with medical reference librarians with expertise in systematic reviews (Supplemental Material, available online at http://www.mayoclinicproceedings.org). This product is intended to have minimal manual processing. Search output is categorized into topic areas (eg, prevention, diagnosis, treatment, and prognosis) and is prioritized by highlighting the studies expected to impact or change practice and placing them at the top of the list. We arbitrarily assumed that such studies are those with any of these characteristics: more than 100 participants, randomized, systematic review or a guideline, or published in the 10 journals with highest impact factor.

       Repository of Studies

      When 3 days have passed on the “What is New?” list of studies, a new list is generated. The expired list is entered in a database that contains all studies about COVID-19. Consequently, this database would contain all studies published since the date of the first case report of COVID-19. This database is critical for conducting any future systematic or rapid review because reviewers would only have to search this database for studies that fit their new question, as opposed to designing and executing searches in bibliographic databases.

       Rapid Reviews

      Frontline clinicians and health system guideline developers provided questions about practice dilemmas they were facing during the pandemic. The program conducted rapid reviews to answer these questions with turnaround time of 3 to 4 days. The reviews emphasized speed and brevity. Steps that deviated from a standard systematic review process were searching a single database, restricting to English language, heavily depending on general search engines such as Google, selecting and extracting studies by a single reviewer, and rarely requiring meta-analysis.

       Repository of Reviews

      This repository included systematic and rapid reviews identified through the following sources: rapid reviews about COVID-19 conducted by the Mayo program, reviews identified through the twice-weekly search, reviews identified through a dedicated overview of systematic reviews search (Supplemental Material), retrieved from a discrete list of websites (Supplemental Material), or identified through networking with other academic evidence synthesis programs. Some reviews included indirect evidence derived from studies about other coronaviruses or respiratory infections, when appropriate. Reviews were indexed in population, intervention, comparison, outcome (PICO) format with quantitative or narrative summary of the main outcomes and also indexed according to the same topic areas. Grading of the certainty in the evidence, a critical component of any synthesis,
      • Murad M.H.
      Clinical practice guidelines: a primer on development and dissemination.
      was extracted from the existing review if available. A final and rigorous grading process is planned when feasible from a time perspective.

      Target Audience and Dissemination

      Identifying a target audience aside from the stakeholders who requested a particular review was challenging in a complex health system. Mass email strategy was considered ineffective and not pursued. We eventually followed the approach of identifying key partners working on developing guidelines and institutional policies and procedures, and we created an internal website with all the data available without the need for a password. Another important audience was researchers working on various aspects of COVID-19 such as conducting trials or developing systematic review protocols.

      Discussion

      We describe the development and implementation of a framework for evidence synthesis programs during the COVID-19 pandemic. This framework was developed using rapid cycles of iterative components and was found feasible to implement in a large multistate health system. After 1 month of implementation, this team has conducted seven rapid reviews and indexed more than 100 reviews; the database contains approximately 2000 studies. This framework is likely generalizable to the extent that it can be activated in other future pandemics or urgent crises such as large-scale novel environmental exposures.

      Study Limitations

      Developing a list of studies published in the previous 3 days is surprisingly difficult and requires a manual process. Varying filters in databases do not function well and confuse electronic publication dates with print dates. The prioritization of studies scheme that we used clearly can introduce publication bias, particularly when focusing on specific journals. Existing systematic review registries (eg, the International Prospective Register of Systematic Reviews) were neither up-to-date nor helpful. A major disincentive to conducting rapid reviews is the difficulty in publishing them by peer-reviewed journals. Rapid reviews themselves have important limitations. For example, not duplicating the process of study selection by two independent reviewers can introduce an error rate of approximately 11%.
      • Wang Z.
      • Nayfeh T.
      • Tetzlaff J.
      • O'Blenis P.
      • Murad M.H.
      Error rates of human reviewers during abstract screening in systematic reviews.
      Within weeks of the pandemic, multiple systematic reviews had been published about the same topic, which led to a challenge of determining which review to use, which one is most recent, and the extent of overlap of included studies across reviews. Another limitation relates to any new or novel topic: earlier studies likely exaggerate the treatment effects, which is known as the Proteus phenomenon.
      • Wang Z.
      • Alahdab F.
      • Almasri J.
      • et al.
      Early studies reported extreme findings with large variability: a meta-epidemiologic study in the field of endocrinology.
      ,
      • Alahdab F.
      • Farah W.
      • Almasri J.
      • et al.
      Treatment Effect in Earlier Trials of Patients With Chronic Medical Conditions: A Meta-Epidemiologic Study.
      Extrapolation from studies about other corona viruses or respiratory infections can lead to indirect evidence and lower certainty.

      Conclusion

      We present a framework for evidence synthesis programs to respond to pandemics that has proven feasible and practical during the response to COVID-19. Whereas some health systems may not have a similar infrastructure for timely implementation of rapid evidence synthesis activities, collaboration across health systems is critical so that emerging knowledge is efficiently shared across systems and globally.

      Supplemental Online Material

      References

        • Shah A.
        • Kashyap R.
        • Tosh P.
        • Sampathkumar P.
        • O'Horo J.C.
        Guide to understanding the 2019 novel coronavirus.
        Mayo Clin Proc. 2020; 95: 646-652
        • Smith V.
        • Devane D.
        • Begley C.M.
        • Clarke M.
        Methodology in conducting a systematic review of systematic reviews of healthcare interventions.
        BMC Med Res Methodol. 2011; 11: 15
        • Schoelles K.
        • Umscheid C.A.
        • Lin J.S.
        • et al.
        A Framework for Conceptualizing Evidence Needs of Health Systems. Agency for Healthcare Research and Quality, Rockville, MD2017
        • Haby M.M.
        • Chapman E.
        • Clark R.
        • Barreto J.
        • Reveiz L.
        • Lavis J.N.
        What are the best methodologies for rapid reviews of the research evidence for evidence-informed decision making in health policy and practice: a rapid review.
        Health Res Policy Syst. 2016; 14: 83
        • Murad M.H.
        Clinical practice guidelines: a primer on development and dissemination.
        Mayo Clin Proc. 2017; 92: 423-433
        • Wang Z.
        • Nayfeh T.
        • Tetzlaff J.
        • O'Blenis P.
        • Murad M.H.
        Error rates of human reviewers during abstract screening in systematic reviews.
        PLoS One. 2020; 15: e0227742
        • Wang Z.
        • Alahdab F.
        • Almasri J.
        • et al.
        Early studies reported extreme findings with large variability: a meta-epidemiologic study in the field of endocrinology.
        J Clin Epidemiol. 2016; 72: 27-32
        • Alahdab F.
        • Farah W.
        • Almasri J.
        • et al.
        Treatment Effect in Earlier Trials of Patients With Chronic Medical Conditions: A Meta-Epidemiologic Study.
        Mayo Clinic Proc. 2018; 93: 278-283