Advertisement
Mayo Clinic Proceedings Home

How Many Contemporary Medical Practices Are Worse Than Doing Nothing or Doing Less?

  • John P.A. Ioannidis
    Correspondence
    Correspondence: Address to John P. A. Ioannidis, MD, DSc, Stanford Prevention Research Center, Medical School Office Bldg Rm X306, 1265 Welch Rd, Stanford, CA 94305.
    Affiliations
    Stanford Prevention Research Center, Department of Medicine, Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA
    Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA
    Search for articles by this author
      How many contemporary medical practices are not any better than or are worse than doing nothing or doing something else that is simpler or less expensive? This is an important question, given the negative repercussions for patients and the health care system of continuing to endorse futile, inefficient, expensive, or harmful interventions, tests, or management strategies. In this issue of Mayo Clinic Proceedings, Prasad et al
      • Prasad V.
      • Vandross A.
      • Toomey C.
      • et al.
      A decade of reversal: an analysis of 146 contradicted medical practices.
      describe the frequency and spectrum of medical reversals determined from a review of all the articles published over a decade (2001-2010) in New England Journal of Medicine (NEJM). Their work extends a previous effort
      • Prasad V.
      • Gall V.
      • Cifu A.
      The frequency of medical reversal.
      that had focused on data from a single year and had suggested that almost half of the established medical practices that are tested are found to be no better than a less expensive, simpler, or easier therapy or approach. The results from the current larger sample of articles
      • Prasad V.
      • Vandross A.
      • Toomey C.
      • et al.
      A decade of reversal: an analysis of 146 contradicted medical practices.
      are consistent with the earlier estimates: 27% of the original articles relevant to medical practices published in NEJM over this decade pertained to testing established practices. Among them, reversal and reaffirmation studies were approximately equally common (40.2% vs 38%). About two-thirds of the medical reversals were recommended on the basis of randomized trials. Even though no effort was made to evaluate systematically all evidence on the same topic (eg, meta-analyses including all studies published before and after the specific NEJM articles), the proportion of medical reversals seems alarmingly high. At a minimum, it poses major questions about the validity and clinical utility of a sizeable portion of everyday medical care.
      Are these figures representative of the medical literature and evidence base at large? The sample assembled by Prasad et al is highly impressive, but it accounts for less than 1% of all randomized trials published in the same decade (an estimated >10,000 per year) and an even more infinitesimal portion of other types of study designs. If one could extrapolate from this sample by proportion, perhaps there have been several tens of thousands of medical reversal studies across all 23 million articles entered to date in PubMed. One has to be cautious with extrapolations, however. New England Journal of Medicine is clearly different from other journals in many ways besides having the highest impact factor among the list of 155 general and internal medicine journals.

      ISI Web of Science. Journal Citation Reports. Accessed May 9, 2013.

      It is widely read, and it has high visibility and impact both on the mass media and on medical practitioners. In this regard, the collection of 146 medical reversals reviewed by Prasad et al is a compendium of widely known, visible examples, and thus it can make excellent reading for medical practitioners and researchers, teachers, and trainees. At the same time, this characteristic is also a disadvantage: the articles published by NEJM are a highly selected sample, probably susceptible to publication and selective outcome reporting bias. There is substantial empirical evidence that the effect sizes of randomized trials published in NEJM, Lancet, or JAMA (the top 3 general and internal medicine journals in terms of impact factor

      ISI Web of Science. Journal Citation Reports. Accessed May 9, 2013.

      ) are markedly inflated, in particular for small trials
      • Siontis K.C.
      • Evangelou E.
      • Ioannidis J.P.
      Magnitude of effects in clinical trials published in high-impact general medical journals.
      ; conversely, the effect sizes for large trials are similar to those seen in large trials on the same topic in other journals.
      • Siontis K.C.
      • Evangelou E.
      • Ioannidis J.P.
      Magnitude of effects in clinical trials published in high-impact general medical journals.
      The interpretation of the results in NEJM is also likely to be more exaggerated compared with other journals because authors may feel pressured to claim that the results are impressive in order to get their work published in such a competitive venue.
      • Young N.S.
      • Ioannidis J.P.
      • Al-Ubaydli O.
      Why current publication practices may distort science.
      Finally, when the quantitative data on effect sizes are examined, studies published in NEJM and other major journals have higher informativity (information gain or change in entropy),
      • Evangelou E.
      • Siontis K.C.
      • Pfeiffer T.
      • Ioannidis J.P.
      Perceived information gain from randomized trials correlates with publication in high-impact factor journals.
      ie, their results do change previous evidence more than the change incurred by the results of studies published elsewhere.
      On the basis of these considerations, the frequency of medical reversals published in NEJM may be somewhat higher than what might be seen in publications in other journals. However, there are also some other counterbalancing forces that could cause bias in the opposite direction. For example, evaluations published in NEJM are likely to focus on commonly used, established medical practices. Such commonly used practices are likely to have had at least some previous evidence generated in the past supporting their use. Conversely, established interventions that are more narrowly applied and specialized (eg, those for which randomized trials might be published in small-circulation, highly specialized journals) may have been originally endorsed with even more sparse and worse-quality evidence, or even no evidence at all.
      Other empirical approaches may also offer some insight about how commonly useless or even harmful treatments are endorsed. The Cochrane Database of Systematic Reviews has assembled considerable current medical evidence from clinical trials on diverse interventions. An empirical evaluation of Cochrane reviews in 2004 showed that most (47.8%) concluded that there is insufficient evidence to endorse the examined interventions.
      • El Dib R.P.
      • Atallah A.N.
      • Andriolo R.B.
      Mapping the Cochrane evidence for decision making in health care.
      A repeated evaluation in 2011 showed that this trend has not changed, with the percentage of insufficient evidence remaining as high as 45%.

      Villas Boas PJ, Spagnuolo RS, Kamegasawa A, et al. Systematic reviews showed insufficient evidence for clinical practice in 2004: what about in 2011? the next appeal for the evidence-based medicine age [published online ahead of print July 3, 2012]. J Eval Clin Pract. http://dx.doi.org/10.1111/j.1365-2753.2012.01877.x.

      Often, non-Cochrane reviews tend to have more positive conclusions about the assessed interventions, but it is unclear whether this finding reflects genuine superiority of the assessed interventions or bias in the interpretation of the results.
      • Tricco A.C.
      • Tetzlaff J.
      • Pham B.
      • Brehaut J.
      • Moher D.
      Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study.
      Although a substantial proportion of interventions are clearly harmful or inferior to others, many are still being used because of reluctance or resistance to abandoning them.
      • Prasad V.
      • Cifu A.
      • Ioannidis J.P.
      Reversals of established medical practices: evidence to abandon ship.
      Some are even widely used despite the poor evidence, as Prasad et al
      • Prasad V.
      • Vandross A.
      • Toomey C.
      • et al.
      A decade of reversal: an analysis of 146 contradicted medical practices.
      eagerly highlight with several examples. Moreover, different medical specialties may vary in their lack of evidence—eg, primary care, surgery, and dermatology interventions more frequently lack evidence to support their use compared with internal medicine interventions.
      • Matzen P.
      How evidence-based is medicine? A systematic literature review.
      Most new interventions that are successfully introduced into medical care have small effects that translate to modest, incremental benefits.
      • Djulbegovic B.
      • Kumar A.
      • Glasziou P.P.
      • et al.
      New treatments compared to established treatments in randomized trials.
      Empirical evaluations have suggested that well-validated large benefits for measurable outcomes such as mortality are uncommon in medicine.
      • Pereira T.V.
      • Horwitz R.I.
      • Ioannidis J.P.
      Empirical evaluation of very large treatment effects of medical interventions.
      Under these circumstances, even subtle changes in the composition and spectrum of the treated population over time, emergence of previously unrecognized toxicities, or a relatively disadvantageous cost can easily tip the evidence balance against the use of these interventions. Moreover, the introduction of interventions with limited or no evidence of benefit continues at fast pace even in specialties that have a strong tradition of evidence-based methods. For example, in almost half (48%) of the recommendations in major cardiology guidelines, the level of evidence is grade C, ie, limited evidence and expert opinion have a highly influential presence.
      • Tricoci P.
      • Allen J.M.
      • Kramer J.M.
      • Califf R.M.
      • Smith Jr., S.C.
      Scientific evidence underlying the ACC/AHA clinical practice guidelines.
      Once we divert beyond traditional treatments (eg, drugs or devices) to diagnostic tools, prognostic markers, health systems, and other health care measures, randomized trials are a rarity.
      • Ferrante di Ruffano L.
      • Davenport C.
      • Eisinga A.
      • Hyde C.
      • Deeks J.J.
      A capture-recapture analysis demonstrated that randomized controlled trials evaluating the impact of diagnostic tests on patient outcomes are rare.
      For example, it has been estimated that, on average, there are only 37 publications per year of randomized trials assessing the effectiveness of diagnostic tests.
      • Ferrante di Ruffano L.
      • Davenport C.
      • Eisinga A.
      • Hyde C.
      • Deeks J.J.
      A capture-recapture analysis demonstrated that randomized controlled trials evaluating the impact of diagnostic tests on patient outcomes are rare.
      Some modern technologies (eg, “omics”) promise to introduce new tools into medical management at such a high pace that many investigators are wary of even thinking about the possibility of randomized testing. Despite better laboratory science, fascinating technology, and theoretically mature designs after 65 years of randomized trials, ineffective, harmful, expensive medical practices are being introduced more frequently now than at any other time in the history of medicine. Under the current mode of evidence collection, most of these new practices may never be challenged.
      The data collected by Prasad et al
      • Prasad V.
      • Vandross A.
      • Toomey C.
      • et al.
      A decade of reversal: an analysis of 146 contradicted medical practices.
      offer some hints about how this dreadful scenario might be aborted. The 146 medical reversals that they have assembled are, in a sense, examples of success stories that can inspire the astute clinician and clinical investigator to challenge the status quo and realize that doing less is more.
      • Grady D.
      • Redberg R.F.
      Less is more: how less health care can result in better health.
      It is not with irony that I call these disasters “success stories.” If we can learn from them, these seemingly disappointing results may be extremely helpful in curtailing harms to patients and cost to the health care system. Although it is important to promote effective practices (“positive success stories”), it is also important to promote and disseminate knowledge about ineffective practices that should be reversed and abandoned. Also, research is needed to find the most efficient ways of applying the knowledge learned from these “negative” studies. Does it suffice to compile lists of practices that should be abandoned?10 What types of educational approaches and reinforcement could enhance their abandonment? What are the obstacles (commercial, professional, system inertia, or other) that hinder this disimplementation step and how can they be best overcome? Are there some incentives that we can offer to practitioners and health systems to apply this “negative” knowledge toward simplifying and streamlining their practices?
      Some of the messaging may require inclusion in guidelines, given the widespread attention that these documents gain, particularly when issued by authoritative individuals or groups, and their capacity to affect clinical practice. Should we require generally higher levels of evidence before practice guidelines are recommended? Moreover, if and when practice guidelines are discredited or overturned by additional information, should notification of practitioners and the public not be undertaken with the same, if not more, vigor as when the practices were first recommended?
      Finally, are there incentives and anything else we can do to promote testing of seemingly established practices and identification of more practices that need to be abandoned? Obviously, such an undertaking will require commitment to a rigorous clinical research agenda in a time of restricted budgets. However, it is clear that carefully designed trials on expensive practices may have a very favorable value of information, and they would be excellent investments toward curtailing the irrational cost of ineffective health care.

      References

        • Prasad V.
        • Vandross A.
        • Toomey C.
        • et al.
        A decade of reversal: an analysis of 146 contradicted medical practices.
        Mayo Clin Proc. 2013; 88: 790-798
        • Prasad V.
        • Gall V.
        • Cifu A.
        The frequency of medical reversal.
        Arch Intern Med. 2011; 171: 1675-1676
      1. ISI Web of Science. Journal Citation Reports. Accessed May 9, 2013.

        • Siontis K.C.
        • Evangelou E.
        • Ioannidis J.P.
        Magnitude of effects in clinical trials published in high-impact general medical journals.
        Int J Epidemiol. 2011; 40: 1280-1291
        • Young N.S.
        • Ioannidis J.P.
        • Al-Ubaydli O.
        Why current publication practices may distort science.
        PLoS Med. 2008; 5: e201
        • Evangelou E.
        • Siontis K.C.
        • Pfeiffer T.
        • Ioannidis J.P.
        Perceived information gain from randomized trials correlates with publication in high-impact factor journals.
        J Clin Epidemiol. 2012; 65: 1274-1281
        • El Dib R.P.
        • Atallah A.N.
        • Andriolo R.B.
        Mapping the Cochrane evidence for decision making in health care.
        J Eval Clin Pract. 2007; 13: 689-692
      2. Villas Boas PJ, Spagnuolo RS, Kamegasawa A, et al. Systematic reviews showed insufficient evidence for clinical practice in 2004: what about in 2011? the next appeal for the evidence-based medicine age [published online ahead of print July 3, 2012]. J Eval Clin Pract. http://dx.doi.org/10.1111/j.1365-2753.2012.01877.x.

        • Tricco A.C.
        • Tetzlaff J.
        • Pham B.
        • Brehaut J.
        • Moher D.
        Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study.
        J Clin Epidemiol. 2009; 62: 380-386.e1
        • Prasad V.
        • Cifu A.
        • Ioannidis J.P.
        Reversals of established medical practices: evidence to abandon ship.
        JAMA. 2012; 307: 37-38
        • Matzen P.
        How evidence-based is medicine? A systematic literature review.
        Ugeskr Laeger. 2003; 165 ([in Danish]): 1431-1435
        • Djulbegovic B.
        • Kumar A.
        • Glasziou P.P.
        • et al.
        New treatments compared to established treatments in randomized trials.
        Cochrane Database Syst Rev. 2012; 10: MR000024
        • Pereira T.V.
        • Horwitz R.I.
        • Ioannidis J.P.
        Empirical evaluation of very large treatment effects of medical interventions.
        JAMA. 2012; 308: 1676-1684
        • Tricoci P.
        • Allen J.M.
        • Kramer J.M.
        • Califf R.M.
        • Smith Jr., S.C.
        Scientific evidence underlying the ACC/AHA clinical practice guidelines.
        JAMA. 2009; 301 ([published correction appears in JAMA. 2009;301(15):1544]): 831-841
        • Ferrante di Ruffano L.
        • Davenport C.
        • Eisinga A.
        • Hyde C.
        • Deeks J.J.
        A capture-recapture analysis demonstrated that randomized controlled trials evaluating the impact of diagnostic tests on patient outcomes are rare.
        J Clin Epidemiol. 2012; 65: 282-287
        • Grady D.
        • Redberg R.F.
        Less is more: how less health care can result in better health.
        Arch Intern Med. 2010; 170: 749-750

      Linked Article

      • Reversal of Medical Practices
        Mayo Clinic ProceedingsVol. 88Issue 10
        • Preview
          We recently read with great interest “A Decade of Reversal: An Analysis of 146 Contradicted Medical Practices” by Prasad et al,1 along with the accompanying editorial by Ioannidis.2 The authors have done an admirable job of quantitatively analyzing the number of reversals published over a decade in one high-impact journal, of course begging the question of how many reversals might be found across the medical literature. We would like to raise 2 important points. The first is about dissemination of the finding of reversal.
        • Full-Text
        • PDF