Advertisement
Mayo Clinic Proceedings Home

Users’ Guide to Medical Decision Analysis

  • Claudia C. Dobler
    Correspondence
    Correspondence: Address to Claudia C. Dobler, MD, PhD, Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Queensland, Australia.
    Affiliations
    Institute for Evidence-Based Healthcare, Faculty of Health Sciences & Medicine, Bond University, Gold Coast, Queensland, Australia

    Evidence-Based Practice Center, Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN
    Search for articles by this author
  • Gordon H. Guyatt
    Affiliations
    Department of Medicine and Department of Health Research Methods, Evidence & Impact, McMaster University, Hamilton, Ontario, Canada
    Search for articles by this author
  • Zhen Wang
    Affiliations
    Institute for Evidence-Based Healthcare, Faculty of Health Sciences & Medicine, Bond University, Gold Coast, Queensland, Australia
    Search for articles by this author
  • M. Hassan Murad
    Affiliations
    Institute for Evidence-Based Healthcare, Faculty of Health Sciences & Medicine, Bond University, Gold Coast, Queensland, Australia
    Search for articles by this author

      Abstract

      Clinicians regularly have to trade benefits and harms to choose between testing and treatment strategies. This process is often done by making global and implicit judgments. A decision analysis is an analytic method that makes this process more explicit, reproducible, and evidence-based. While clinicians are unlikely to conduct their own decision analysis, they will read publications of such analyses or use guidelines based on them. This review outlines the anatomy of a decision tree and provides clinicians with the tools to critically appraise a decision analysis and apply its results to medical decision making. Clinicians reading about a decision analysis can make two judgments. The first judgment is about the credibility of the methods, such as whether the decision analysis addressed a relevant clinical question, included all important outcomes, used the current best evidence to derive variables in the model, and adopted the appropriate time horizon. The second judgment is about rating confidence in the preferred course of action by determining the certainty in the model variables, whether the results are robust in sensitivity analyses and if the results are applicable to a specific patient. Results from a valid and robust decision analysis can inform both guideline panels and the patient-clinician dyad engaged in shared decision-making.

      Abbreviations and Acronyms:

      ICER (incremental cost-effectiveness ratio), IGRA (interferon-gamma release assay test), QALY (quality-adjusted life year), TB (tuberculosis), TST (tuberculin skin test)
      Article Highlights
      • This guide provides clinicians with the tools to critically appraise a decision analysis and apply its results to medical decision making.
      • We describe how clinicians can judge the credibility of the methods of a decision analysis and rate the confidence in the preferred course of action.
      • The outlined principles can also be applied to appraise a cost-effectiveness analysis.
      • We use a clinical scenario and analyze a published decision analysis to demonstrate the practical application of this guide.

      Clinical Scenario

      You are a family physician who sees a 54-year old man, born in India, who has lived in the United States for the past 3 years and was recently diagnosed with diabetes mellitus. You are aware that India has a high incidence of tuberculosis and therefore your patient could be infected with Mycobacterium tuberculosis (latent tuberculosis [TB] infection). Moreover, you are aware that patients with diabetes have a two- to four-fold increased risk of developing active tuberculosis.
      • Al-Rifai R.H.
      • Pearson F.
      • Critchley J.A.
      • Abu-Raddad L.J.
      Association between diabetes mellitus and active tuberculosis: a systematic review and meta-analysis.
      You wonder if you should test your patient for latent TB using an interferon-gamma release assay test (IGRA) or a tuberculin skin test (TST) and prescribe preventive treatment if the test is positive.
      To inform your discussion, you search first for an evidence-based recommendation and find a guideline indicating that there is only low-quality evidence addressing latent TB screening and treatment in immigrants from high-TB-burden countries.
      The same guideline concludes that the benefits of systematic and routine latent TB testing and treatment in patients with diabetes “do not outweigh the risks unless individuals/patients also belong to the groups mentioned in the above recommendations [including immigrants from high burden countries].”
      As your patient is an immigrant from a high-TB-burden country and also has diabetes, you conclude that your patient might potentially benefit from latent TB testing and treatment. However, because the evidence is low quality, you explore further and find a decision analysis with cost-effectiveness measures addressing latent TB testing and treatment in residents born outside the United States with and without medical comorbidities.
      • Tasillo A.
      • Salomon J.A.
      • Trikalinos T.A.
      • Horsburgh Jr., C.R.
      • Marks S.M.
      • Linas B.P.
      Cost-effectiveness of testing and treatment for latent tuberculosis infection in residents born outside the United States with and without medical comorbidities in a simulation model.
      You ascertain that there is reasonable match between the decision problem examined in the analysis and your clinical question. How should you use this decision analysis/cost-effectiveness analysis to help guide you and your patient in deciding whether to pursue the screening option?

      Introduction

      Medical decisions are often complex, and clinicians must regularly weigh the pros and cons of interventions in individual patients. Even treatments that have been found to be effective in large trials are associated with adverse events, and it may not always be apparent whether the benefits (in this case reduced risk of developing TB with treatment of latent TB) outweigh the potential harms (here, receiving treatment based on a false-positive test result, or having an adverse effect, such as drug-induced hepatitis, from latent TB treatment).
      Clinicians often assess such risk-benefit decisions based on their intuition and often make decisions based on mental shortcuts, also called heuristics.
      • Tversky A.
      • Kahneman D.
      Judgment under uncertainty: heuristics and biases.
      This form of intuitive decision-making, although often helpful, can lead to inaccurate assessment of the likely harms and benefits of an intervention.
      Decision analysis addresses medical decision-making by providing a systematic approach to evaluating alternative strategies using objective evidence and careful, model-based evaluation. High-quality medical decision analyses and cost-effectiveness analyses provide decision makers — both clinicians and their patients as well as those involved in health policy — with guidance regarding complex decision problems. Although clinicians will seldom construct their own decision trees when faced with a clinical conundrum, the insights they gain from understanding medical decision analysis will help them to critically reflect on their own decision making.
      The purpose of this users’ guide is to introduce the fundamental concepts of decision analysis and cost-effectiveness analysis and provide users with the tools to appraise studies using these analyses. In-depth descriptions of decision modeling techniques can be found elsewhere.
      • Caro J.J.
      • Briggs A.H.
      • Siebert U.
      • Kuntz K.M.
      Force I-SMGRPT
      Modeling good research practices—overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—1.
      • Roberts M.
      • Russell L.B.
      • Paltiel A.D.
      • et al.
      Conceptualizing a model: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—2.
      • Siebert U.
      • Alagoz O.
      • Bayoumi A.M.
      • et al.
      State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—3.
      • Karnon J.
      • Stahl J.
      • Brennan A.
      • et al.
      Modeling using discrete event simulation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—4.
      • Pitman R.
      • Fisman D.
      • Zaric G.S.
      • et al.
      Dynamic transmission modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--5.
      • Briggs A.H.
      • Weinstein M.C.
      • Fenwick E.A.L.
      • et al.
      Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—6.
      • Eddy D.M.
      • Hollingworth W.
      • Caro J.J.
      • et al.
      Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—7.
      In 1995, Richardson et al
      • Richardson W.S.
      • Detsky A.S.
      Users' guides to the medical literature. VII. How to use a clinical decision analysis. A. Are the results of the study valid? Evidence-Based Medicine Working Group.
      ,
      • Richardson W.S.
      • Detsky A.S.
      Users' guides to the medical literature. VII. How to use a clinical decision analysis. B. What are the results and will they help me in caring for my patients? Evidence Based Medicine Working Group.
      developed a users’ guide (in two parts) on how to use a “clinical decision analysis” and presented a framework for critical appraisal of clinical decision analyses.
      • Richardson W.S.
      • Detsky A.S.
      Users' guides to the medical literature. VII. How to use a clinical decision analysis. A. Are the results of the study valid? Evidence-Based Medicine Working Group.
      ,
      • Richardson W.S.
      • Detsky A.S.
      Users' guides to the medical literature. VII. How to use a clinical decision analysis. B. What are the results and will they help me in caring for my patients? Evidence Based Medicine Working Group.
      Since then, there has been a proliferation of medical decision analysis studies, in particular cost-effectiveness analyses. Markov or state-transition models, not explained in the original users’ guide, are now included in most decision analyses. This current guide presents a contemporary conceptualization and update on decision analysis. It is structured using the following section headings: “Are the Results Valid?” “What Are the Results?” and “How Do I Apply the Results to My Patient?” Before explicating the criteria for using a decision analysis, we will comment on when such an analysis might be helpful and present its anatomy.
      Branch and node decision trees and state-transition models, which are explained in this guide, are “basic” decision model techniques. There are numerous other modeling techniques that are used to predict the events that occur after a clinical or policy decision is made.

      When Might Medical Decision Analysis Be Helpful?

      In clinical practice it may not always be clear whether the benefits of an intervention outweigh the potential harms. Therefore, there is often uncertainty regarding the best course of action, even when randomized controlled trials have shown an intervention’s effectiveness. For example, prophylactic anticoagulation in a hospitalized patient reduces the risk of venous thromboembolism but increases the risk of a major bleeding event. The best course of action in this situation will depend on patients’ risk factors for venous thromboembolism, bleeding risk, and possible contraindications to anticoagulation.
      • Qaseem A.
      • Chou R.
      • Humphrey L.L.
      • Starkey M.
      • Shekelle P.
      Venous thromboembolism prophylaxis in hospitalized patients: a clinical practice guideline from the American College of Physicians.
      ,
      • Grant P.J.
      • Conlon A.
      • Chopra V.
      • Flanders S.A.
      Use of venous thromboembolism prophylaxis in hospitalized patients.
      Medical decision analysis helps to determine the best course of action when there is a trade-off between different management strategies.
      Although the risk-benefit trade-off commonly refers to clinical benefits and harms, it can also include the trade-off between greater effectiveness and higher cost. In this case, the appropriate decision model is a cost-effectiveness analysis that compares the relative costs and values (often expressed as per quality-adjusted life year [QALY]) of different courses of action). Cost-effectiveness analyses that address patient-important effectiveness outcomes can inform clinical decisions as well as health policy decisions.

      The Anatomy of a Decision Analysis Based on a Decision Tree

      Table 1 summarizes the typical process of a decision analysis based on a decision tree, and Figure 1A shows an example of a decision tree that lies at the heart of decision analysis. Decision trees include a decision node (represented by a square) and a number of decision branches (Figure 1A). Every decision branch is followed by further branches representing the occurrence or non-occurrence of relevant events that may follow from the management decision (eg, developing drug-induced hepatitis from latent TB treatment, dying from drug-induced hepatitis, or dying from an unrelated cause). These events are represented by chance nodes (circles) with the associated probability for alternative outcomes at that node. Terminal nodes (triangles) show the final outcome of a decision path.
      Table 1The Process of Conducting a Decision Analysis
      1. Formulate the question for a specific group of patients using an actionable intervention and a clear alternative course of action considering outcomes that are relevant for the decision at hand (ie, participants, intervention, comparator, outcome, aka the PICO process).
      2. Construct a decision tree with decision branches for potential courses of action (at least two) followed by chance nodes
      Chance nodes are typically represented by a circle; chance nodes are steps in the decision tree that involve uncertainties. The possible outcomes of these probabilistic events are not under the control of the decision maker and are shown as paths leading away from the node toward the right.
      representing the occurrence or non-occurrence of relevant outcomes
      3. Determine if a Markov model
      The Markov model is a stochastic model that shows uncertain events as transitions between defined health states and the rate of transitions and probabilities between the states. Markov models are useful to model risk over time, timing of events, and events that may happen more than once.
      should be added to the decision tree, which is desirable if the decision problem involves a risk that continuous over time.
      4. If using a Markov model,
      The Markov model is a stochastic model that shows uncertain events as transitions between defined health states and the rate of transitions and probabilities between the states. Markov models are useful to model risk over time, timing of events, and events that may happen more than once.
      choose an appropriate model time horizon and Markov cycle length.
      5. For a cost-effectiveness analysis, choose the desired analysis perspective (eg, from a health care provider view, societal view, etc).
      6. Identify the probabilities associated with chance nodes
      Chance nodes are typically represented by a circle; chance nodes are steps in the decision tree that involve uncertainties. The possible outcomes of these probabilistic events are not under the control of the decision maker and are shown as paths leading away from the node toward the right.
      and the utilities or values associated with the outcomes.
      7. Conduct a literature search that inform the probabilities and utilities.
      8. Determine the preferred course of action by “folding back”
      “Folding back” the decision tree analysis method indicates working backwards. One begins at a far right chance node of the decision tree. At every chance node the utilities from all chance branches to the right are summed up by multiplying the probability assigned to every chance branch (providing the weight to derive the expected value) with its associated utility and adding the results from all branches.
      the decision tree.
      9. Conduct a sensitivity analysis that informs the preferred course of action.
      a Chance nodes are typically represented by a circle; chance nodes are steps in the decision tree that involve uncertainties. The possible outcomes of these probabilistic events are not under the control of the decision maker and are shown as paths leading away from the node toward the right.
      b The Markov model is a stochastic model that shows uncertain events as transitions between defined health states and the rate of transitions and probabilities between the states. Markov models are useful to model risk over time, timing of events, and events that may happen more than once.
      c “Folding back” the decision tree analysis method indicates working backwards. One begins at a far right chance node of the decision tree. At every chance node the utilities from all chance branches to the right are summed up by multiplying the probability assigned to every chance branch (providing the weight to derive the expected value) with its associated utility and adding the results from all branches.
      Figure thumbnail gr1a
      Figure 1A, Example of decision tree structure. B, Markov sub-tree. A simplified decision tree depicts two courses of action: no testing/treatment and interferon-gamma release assay (IGRA) and treatment for positive IGRA that impact outcomes associated with the presence or absence of latent tuberculosis (TB).
      Figure thumbnail gr1b
      Figure 1A, Example of decision tree structure. B, Markov sub-tree. A simplified decision tree depicts two courses of action: no testing/treatment and interferon-gamma release assay (IGRA) and treatment for positive IGRA that impact outcomes associated with the presence or absence of latent tuberculosis (TB).
      Most outcomes occur over varying time frames, and patients may experience more than one outcome over time. An optimal decision analysis will be able to take the time factor and the probability of a patient experiencing more than one event over time into account. The methodology for this process involves what is called a Markov or state-transition model (Figure 1B), which allows modeling a decision problem that involves a risk that continuous over time and has outcomes that occur at different points in time (eg, ongoing risk over a lifetime of developing TB in a patient with latent TB), and when outcomes of interest may happen more than once (eg, development of a repeat episode of TB).
      • Siebert U.
      • Alagoz O.
      • Bayoumi A.M.
      • et al.
      State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—3.
      A state-transition model consists of cycles of equal increments of time (eg, 1 month or 12 months). One assumption of the model is that in every cycle the patient resides in a finite number of "health states."
      • Siebert U.
      • Alagoz O.
      • Bayoumi A.M.
      • et al.
      State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—3.
      Thus, in the model, a patient may be alive and TB free, develop TB, be cured from TB, die from TB, or die from an unrelated cause (Figure 1B). When moving through cycles, patients transition between these different outcomes (eg, after developing active TB a patient might recover from TB or die of TB). Another assumption is that future outcome only depends on current outcome, not past outcome.
      Beware that figures accompanying analyses in published papers are highly simplified schematics. They will usually not allow the reader to understand the detailed model structure.
      We will now move to our criteria for assessing the credibility and applying a decision analysis (Table 2).
      Table 2Guide for Appraising and Applying the Results of a Decision Analysis
      First Judgment: Evaluate the Credibility of the Methods of a Decision Analysis
       Did the decision analysis address a relevant clinical question including all important outcomes?
       Is the time horizon
      All Markov models include a time horizon, whereas simple decision trees do not. A Markov model should therefore be used when the timing of events is important (eg, the risk of preventive treatment occurs now, but the benefit is expected in the future).
      of the decision analysis appropriate?
       Are variables in the model (probabilities and utilities) based on current best evidence?
       For cost-effectiveness analysis: what was the analysis perspective?
      Second Judgment: Rate the Confidence in the Preferred Course of Action
       What are the sources of the probabilities and utilities in the decision tree?
       How robust are the decision analysis results?
       Are the results applicable to my patient?
      a All Markov models include a time horizon, whereas simple decision trees do not. A Markov model should therefore be used when the timing of events is important (eg, the risk of preventive treatment occurs now, but the benefit is expected in the future).

      First Judgment: Was the Methodology of the Decision Analysis Credible?

      Table 3 summarizes all components of judgment 1 — determining the credibility of the methods of a decision analysis.
      Table 3Using the Guide: Judgment 1, Determining the Credibility of the Methods of a Decision Analysis (Cost-Effectiveness Analysis for Latent TB), applied to the study of Tasillo et al.
      Tasillo et al.3
      The decision analysis addressed a relevant clinical question of a specific group of patients (non-US born residents with and without medical comorbidities). The five evaluated courses of action were clear and actionable in a clinical setting where both tests (tuberculin skin test [TST] and interferon-gamma release assay [IGRA] test) are available. The main outcome quality-adjusted life-years (QALYs) was a patient-important outcome relevant for clinical decision making.
      The clinically most important possible outcomes (even when rare) were included: developing active TB, being cured from TB, dying from TB, experiencing drug-induced hepatitis from latent TB treatment, dying from drug-induced hepatitis, dying from causes other than TB.
      The decision analysis used a Markov model with a time horizon of a lifetime which allowed for appropriately modelling the risk of TB re-activation over a person’s lifetime.
      The decision analysis authors performed a meta-analysis for the diagnostic performance of the TST and IGRA — they did not specify that they followed the procedures of a trustworthy systematic review.
      • Murad M.H.
      • Montori V.M.
      • Ioannidis J.P.
      • et al.
      How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature.
      For the other variable values in the model the authors chose one or two studies to provide estimates. The representativeness of these studies is uncertain. The effectiveness estimate of latent TB treatment (relative risk of developing TB with treatment) was sourced from a randomized trial and the probability of drug-induced hepatitis was sourced from two randomized trials. The probabilities of the risk of developing TB in patients with diabetes and latent TB the case-fatality rate of TB and the drug- case-fatality of drug-induced hepatitis were sourced from one observational study respectively.
      To support their choice of utilities the authors cite two studies, one addressing utilities in liver disease and the other in latent and active TB. The rigor and representativeness of these studies remains uncertain.
      The study was a cost-effectiveness analysis from the health care sector perspective. While the cost estimates are thus not suitable for individual decision making, the effectiveness outcome of the model (QALYs) can be used for decision making in individual patients.
      The authors provided most of the information needed to address confidence in the preferred course of action. They described the sources of the probabilities and utilities in the decision tree although as we have noted the sources have weaknesses. The authors included a general statement that “In general the base case conclusions were robust to changes in core model parameters.” A probabilistic sensitivity analysis using Monte Carlo simulation was conducted for the prevalence of latent TB test characteristics and the rate of developing TB but not for any utility estimates). The model seemed to be robust for supporting “testing and treatment” (based on assessment of QALYs) but not the choice of testing (see Table 5).
      The sensitivity analysis results were provided for the cost-effectiveness (incremental cost-effectiveness ratio = cost/QALY) but not for QALYs alone.
      a Tasillo et al.
      • Tasillo A.
      • Salomon J.A.
      • Trikalinos T.A.
      • Horsburgh Jr., C.R.
      • Marks S.M.
      • Linas B.P.
      Cost-effectiveness of testing and treatment for latent tuberculosis infection in residents born outside the United States with and without medical comorbidities in a simulation model.

      Did the Decision Analysis Address a Relevant Clinical Question including all Important Outcomes?

      Medical decision analyses should have a clear focus 1) addressing a clinical question for a specific group of patients; 2) using an actionable intervention and one or multiple clear alternative course(s) of action; and 3) using outcomes that are relevant for clinical decision making (ie, participants, intervention, comparator, and outcome, aka the PICO process). The decision tree on which the decision analysis is based must include all choice alternatives that are relevant to clinical decision making. Omitting management alternatives can provide very misleading results in a cost-effectiveness analysis (when an effective but less expensive alternative is not included). In our clinical example, relevant alternative strategies include a TST (and treating if there is evidence of infection), or an IGRA (and treating if there is evidence of infection), and no testing and treating.
      A medical decision analysis should include all outcomes that are important to patients (eg, mortality, quality of life, and functional status) rather than surrogate outcomes (eg, result of a laboratory or imaging test). In their efforts to simplify the decision model, decision analysts should not omit rare but important outcomes such as death; any such omission reduces the trustworthiness of the result. Including harms of an intervention is as important as including benefits. For example, a cost-effectiveness analysis of latent TB treatment with isoniazid in close contacts of patients with infectious TB did not include any adverse events of preventive TB treatment and might therefore have overestimated the benefit of preventive TB treatment compared with no treatment.
      • Diel R.
      • Nienhaus A.
      • Schaberg T.
      Cost-effectiveness of isoniazid chemoprevention in close contacts.

      Is the Analytic Time Horizon of the Decision Analysis Appropriate?

      The time frame of the model, often referred to as the time horizon of the analysis, should be long enough to capture any costs and benefits that occur because of the intervention. For some decisions, choosing a lifetime horizon is appropriate, for example, when modeling a latent infection that can develop into active disease at any point during the patient’s life.
      • Dobler C.C.
      • Martin A.
      • Marks G.B.
      Benefit of treatment of latent tuberculosis infection in individual patients.
      A study of the risk of developing TB in young adults with latent TB that only has a time horizon of 20 years
      • Taylor W.C.
      • Aronson M.D.
      • Delbanco T.L.
      Should young adults with a positive tuberculin test take isoniazid?.
      will therefore not capture the cumulative lifetime risk. In other situations it may be appropriate to choose a time frame over a few years, for example, when the decision model focuses on interventions to prevent the recurrence of cancer.
      • Younis T.
      • Rayson D.
      • Skedgel C.
      The cost-utility of adjuvant chemotherapy using docetaxel and cyclophosphamide compared with doxorubicin and cyclophosphamide in breast cancer.

      Are Variables in the Model (Probabilities and Utilities) Based on Current Best Evidence?

      A decision tree is populated with events and health states, each with their associated probabilities (eg, the likelihood of developing active TB) and utilities (eg, a year of life with active TB may be judged as having only half the value of a year of life in full health).
      Every branch in a decision tree is assigned a probability (Figure 2A). Together the probabilities of all branches following a chance node add up to 1. Probabilities that reflect the risk that a certain event will happen are often modified by patient characteristics such as age or gender. It is therefore necessary to specify the population characteristics to which the results of the decision analysis apply.
      Figure thumbnail gr2a
      Figure 2A, Simplified decision tree. B, Simplified “folded back” decision tree. This simplified decision tree (without Markov model, for illustrative purposes only) shows the process of “folding back the tree” to determine the best course of action in somebody who has latent tuberculosis (TB). In this case, the best case of action is to give latent TB treatment with the “no latent TB treatment” alternative marked as the rejected alternative. Total utility for latent TB treatment is 0.9992 versus 0.999 for no latent TB treatment. The outcomes at chance nodes (green) are mutually exclusive, and their combined probability must sum to 1.
      Figure thumbnail gr2b
      Figure 2A, Simplified decision tree. B, Simplified “folded back” decision tree. This simplified decision tree (without Markov model, for illustrative purposes only) shows the process of “folding back the tree” to determine the best course of action in somebody who has latent tuberculosis (TB). In this case, the best case of action is to give latent TB treatment with the “no latent TB treatment” alternative marked as the rejected alternative. Total utility for latent TB treatment is 0.9992 versus 0.999 for no latent TB treatment. The outcomes at chance nodes (green) are mutually exclusive, and their combined probability must sum to 1.
      The combination of probabilities derived from the literature and values (utilities) in a decision model determines the output of the model in terms of the best course of action that maximizes utility.
      • Swales J.D.
      Science in a health service.
      ,
      • Hunink M.
      • Glasziou P.
      • Siegel J.
      • et al.
      Decision Making in Health and Medicine: Integrating Evidence and Values.
      On a utility scale, perfect health is typically assigned a value of 1 and being dead a value of 0. Many health states in a decision model are associated with decrements in quality of life (eg, a patient experiencing an adverse effect of a TB medication has a health state utility of less than 1). Health state utilities enable calculation of QALYs as an outcome. If, for example, a patient lives for 10 years in a health state that is associated with a utility of 0.7, this is equal to 7 QALYs.
      A well-conducted decision analysis should present the sources of all parameters (probabilities, utilities, and cost in a cost-effectiveness analysis; see below) used in the decision tree. Most published decision analyses report the sources of the parameter estimates and some include a systematic review and meta-analysis at least as source for the effectiveness of the intervention(s) under consideration.
      • Malhotra A.
      • Wu X.
      • Forman H.P.
      • Matouk C.C.
      • Gandhi D.
      • Sanelli P.
      Management of tiny unruptured intracranial aneurysms: a comparative effectiveness analysis.
      A critical look at the sources of the probabilities and utilities is perhaps the easiest and most important critical judgement for the clinician user of a decision analysis. Ideally, all important parameters that materially affect the results are based on systematic reviews. However, not all evidence derived from systematic reviews is high quality depending on the quality of the included studies and the review methodology. To the extent that evidence is low quality (that is, associated with high uncertainty), results of the decision analysis will be less robust.

      For Cost-Effectiveness Analysis: What Was the Analysis Perspective?

      Decision analysis can include the expected costs of decision alternatives, referred to as cost-effectiveness analysis. Cost-effectiveness analysis is commonly performed to compare two or more decision alternatives where one offers an improved health outcome but at increased cost.
      • Ryder H.F.
      • McDonough C.
      • Tosteson A.N.
      • Lurie J.D.
      Decision Analysis and Cost-effectiveness Analysis.
      Cost-effectiveness analyses focus on the incremental cost of an intervention for a particular difference in outcome — for instance, the additional cost of a management strategy to delay a single death from TB — referred to as an incremental cost-effectiveness ratio (ICER).
      • Sanders G.D.
      • Neumann P.J.
      • Basu A.
      • et al.
      Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: Second Panel on Cost-Effectiveness in Health and Medicine.
      ,
      Thus, the ICER quantifies the trade-offs between improved health outcomes and resources spent.
      • Bang H.
      • Zhao H.
      Median-based incremental cost-effectiveness ratio (ICER).
      An intervention that is both less effective and more expensive than an alternative intervention is considered to be “strictly dominated” and is clearly inferior.
      • Cohen D.J.
      • Reynolds M.R.
      Interpreting the results of cost-effectiveness studies.
      If there are multiple interventions analyzed, interventions are listed according to cost, with the cheapest and least effective intervention forming the baseline comparison. The ICER for the next-most-expensive intervention is then calculated by dividing the difference in cost by the difference in effectiveness. In this way the analysts can calculate the ICER for all remaining options (Table 4). Dominated interventions are listed but not included in the ICER calculation.
      Table 4Cost-Effectiveness League Table
      In the league table, the possible courses of action are listed in ascending order of their cost. For each course of action the additional (incremental) cost and effectiveness is compared to the previously listed course of action. The incremental cost-effectiveness ratio (ICER) for the tuberculin skin test (TST) strategy is higher than that of the next, more effective, alternative--the interferon-gamma release assay (IGRA) strategy. It is therefore dominated by the combination of 2 alternatives and should not be used to calculate appropriate ICERs. ICER, incremental cost effectiveness ratio; QALY, quality-adjusted life year.
      Course of actionCost, $Incremental cost (per person), $Effectiveness (QALYs)Incremental effectiveness (QALYs)Incremental cost/effectiveness (ICER)
      No testing20 (assumed)70 (assumed)
      Confirm positive674770.00090.000952,200
      TST1023570.00110.0002(175,000) dominated
      IGRA1565470.00160.0005108,000
      Confirm negative2034770.00180.0002235,000
      a In the league table, the possible courses of action are listed in ascending order of their cost. For each course of action the additional (incremental) cost and effectiveness is compared to the previously listed course of action. The incremental cost-effectiveness ratio (ICER) for the tuberculin skin test (TST) strategy is higher than that of the next, more effective, alternative--the interferon-gamma release assay (IGRA) strategy. It is therefore dominated by the combination of 2 alternatives and should not be used to calculate appropriate ICERs. ICER, incremental cost effectiveness ratio; QALY, quality-adjusted life year.
      The unit of effectiveness in a cost-effectiveness analysis is commonly the QALY, which reflects the quantity and quality of life. The use of a utility (perceived health-related quality of life for a health state) allows comparisons across different diseases and health sectors by using a common unit of measure (cost/QALY gained).
      • Dobler C.C.
      Screening strategies for active tuberculosis: focus on cost-effectiveness.
      Benefits, and particularly costs, are likely to differ depending on the perspective of the target audience be it the patient, a hospital, a health maintenance organization, an employer, a government payer, or society. The perspective will determine which costs and health outcomes are included in the model and how they are valued. A cost-effectiveness analysis of an infectious disease from a societal perspective, for example, would include modeling of disease transmission and associated health outcomes in affected people. The most common perspective of cost-effective analyses is the payer perspective, which is not recommended by experts.
      • Sanders G.D.
      • Neumann P.J.
      • Basu A.
      • et al.
      Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: Second Panel on Cost-Effectiveness in Health and Medicine.
      ,
      If the effectiveness results of a cost-effectiveness analysis are used for clinical decision making in individual patients, the model should not include any components that are not relevant to the patient perspective. To the extent that the perspective differs from the one that is relevant to you, you should take that into account. For instance, the individual clinician and patient may be uninterested in system costs. Coming back to the cost-effectiveness analysis in our clinical scenario
      • Tasillo A.
      • Salomon J.A.
      • Trikalinos T.A.
      • Horsburgh Jr., C.R.
      • Marks S.M.
      • Linas B.P.
      Cost-effectiveness of testing and treatment for latent tuberculosis infection in residents born outside the United States with and without medical comorbidities in a simulation model.
      : you would ignore the overall cost estimates, as they are from a health sector rather than a patient perspective, and focus on the health outcomes for decision making in your patient including QALYs and the number of persons needed to test and treat to prevent one case of TB. You might want to add any information on costs that are relevant to the patient, if available. Importantly, as secondary cases from TB transmission are included in this analysis, you would tell your patient that the outcomes also include how other people close to them would be affected by the chosen course.

      Second Judgment: What Is the Confidence in the Preferred Course of Action?

      To determine the preferred course of action from a decision tree, one works backwards (ie, from right to left, that is from the terminal nodes to the initial decision node) by “folding back” the tree. At every chance node the utilities from all chance branches to the right are summed up by multiplying the probability assigned to every chance branch (providing the weight to derive the expected value) with its associated utility and adding the results from all branches. When arriving back at the decision node, the model will present the total utility associated with each option — the one with the highest utility represents the best choice (unless, in a cost-effectiveness analysis, it proves prohibitively expensive). Figure 2B gives an example of the process of folding back a decision tree.

      How Robust Are the Decision Analysis Results?

      To test the impact of uncertainties on the model outcome, the analyst should perform sensitivity analysis — that is, varying probability and utility values through their plausible range — for all key variables in the decision tree. For this purpose, each variable value (for probabilities and utilities) is assigned upper and lower bounds or a distribution reflecting the associated uncertainty. As we have noted, the analyst should base estimates on the best existing evidence from systematic review of the relevant literature. From such reviews will come the upper and lower bounds for the estimates (eg, 95% confidence intervals from a systematic review of primary studies). One-way sensitivity analysis — that is, considering the possible true values of one variable at a time — is commonly used to explore the influence of plausible variation on the decision problem. Analysts can present results of one-way sensitivity analyses in a tornado diagram, which provides information on the relative plausible impact of the variables addressed in sensitivity analyses (see Figure 3, which shows a fictitious example).
      • Dobler C.C.
      • Martin A.
      • Marks G.B.
      Benefit of treatment of latent tuberculosis infection in individual patients.
      Figure thumbnail gr3
      Figure 3Tornado diagram. This is a fictitious example of a tornado diagram for illustrative purposes only. The x-axis of the tornado diagram displays the main outcome (here, difference in quality adjusted life-years by treating latent tuberculosis [TB]). The solid line represents the base case outcome/value, that is, the outcome when the point estimates for all variables (probabilities and utilities) are used. Each variable included in one-way sensitivity analysis has its own bar, and the width of the bar is indicative of the impact that variation of the variable through its plausible range has. Values to the right of the vertical 0-line indicate a net gain and those to the left indicate a net loss in QALYs with latent TB treatment. The bars are arranged in descending order of width, so that the diagram is essentially a ranking of variables based on their potential impact on the outcome of the decision analysis. Special attention should be given to variables that cannot only significantly change the effect size of the outcome but the direction of the overall effect. In the example, the two variables “Risk of developing active TB” and “Quality of life post-TB” can result in different study conclusions (different effect direction) when varied through their plausible range.
      In addition to variable values, the chosen complexity of a model (number of variables included) and the structure of the model (eg, the type of defined health states in a Markov model) can potentially impact the result of a decision analysis. Sensitivity analysis can be used to determine whether the results are robust to changes in model complexity and structure.
      The robustness of the model results is decreased when sensitivity analysis results in important changes in the estimates of effect, most seriously when it changes the direction of the overall effect. If the study conclusion (direction of the effect) changes in the sensitivity analysis of utilities, the preferred course of action is strongly influenced by patients’ preferences.
      In addition to one-way sensitivity analysis, analysts can conduct probabilistic sensitivity analysis using “Monte Carlo simulation.”
      • Sonnenberg F.A.
      • Beck J.R.
      Markov models in medical decision making: a practical guide.
      Here, the analyst can examine the impact of simultaneously varying results of several input variables across their probable range. This process is repeated in numerous simulations (eg, 100,000), and in each simulation the value for each variable is selected at random according to the plausible range or probability distribution. The analyst can then present the proportion of simulations in which one decision option or another was the preferred option.
      We will have more confidence in the result if one decision option is the preferred course in a high number of simulations (eg, in 90% of simulations or more).
      Returning to the clinical scenario (Table 5), you are reasonably confident in the preferred course of action.
      Table 5Using the Guide: Judgment 2, Determining the Confidence in the Preferred Course of Action (Cost-Effectiveness Analysis for Latent Tuberculosis Infection), applied to the study of Tasillo et al
      Tasillo et al.3
      What was the preferred course of action (focusing on effectiveness outcome only)?
       Testing and treatment for latent tuberculosis (TB) always resulted in quality-adjusted life years (QALYs) gained compared with no testing, independently of the testing algorithm used.
       For non-US born patients with diabetes aged 57 years (comparable with our clinical scenario patient), the “confirm negative” testing strategy in which a negative interferon-gamma release assay (IGRA) followed by a positive tuberculin skin test (TST) (or an initial positive IGRA) was interpreted as indicative of latent TB was associated with the highest gain in QALYs compared with no testing (incremental QALY = 0.0018 = 0.7 days) and the lowest number needed to test and treat (n=362). The “confirm positive” strategy, in which a positive TST was followed by an IGRA and both had to be positive to indicate latent TB, had the lowest gain in QALYs compared with no testing (incremental QALY = 0.0009 = 0.3 days) and the highest number needed to test and treat (n=749).
      How robust are the decision analysis results?
       While the robustness of the results could potentially be compromised by concerns regarding sources of both probabilities and utilities, the results were robust in sensitivity analyses. Sensitivity analysis of latent TB prevalence indicated that the prevalence needed to be ≥4.5% in non-US born patients with diabetes (such as our patient in the case scenario) in order for any testing to be the preferred intervention, a criterion that is very likely fulfilled in the patient from India (see below).
       While the preferred testing for latent TB varied in sensitivity analysis for TST specificity, the proportion of patients returning for TST reading, age, and utility (quality of life) with latent TB and post-TB, some form of testing and treatment (as opposed to no testing and treatment) was always the preferred option. Strategies including IGRA were preferred in more than 60% of simulations for non-US born populations with diabetes. For utilities, while certainty in the results was reduced based on the sources of most utilities, the model seemed to be robust for supporting “testing and treatment” but not the choice of testing.
      Do the results apply to my patient?
       While the population assessed in the decision model comprised residents born outside the US with diabetes like our patient, the latent TB prevalence estimate of 15.9% was based on the whole population of non-US born residents, a very heterogeneous group with highly variable estimates for latent TB prevalence depending on the TB incidence in their respective countries of birth. It is estimated that almost 40% of people living in India have latent TB,
      • Chadha V.K.
      Tuberculosis epidemiology in India: a review.
      which means that the pre-test probability in our patient in the case scenario is higher than that of the study population in the decision model. Thus, the benefits of testing/treatment for latent TB compared with no testing are likely greater in our patient compared with the result of the decision analysis. This increases the certainty in the preferred course of action of testing and treating compared with no testing.
      Confidence in the preferred course of action
       The preferred course of action is consistently testing and treating for latent TB with the optimal testing strategy of “confirm negative” showing a QALY gain of 0.7 days compared with no testing and a number needed to test and treat of 362 in non-US born patients with diabetes aged 57 years (comparable to our patient). Based on the robustness of the preferred course of action in sensitivity analysis, indicating that the results are not highly determined by changes to parameter assumptions, we have moderate confidence in the results of the decision analysis despite some concerns regarding sources of included parameters.
       The differences in QALYs between different intervention options including no testing/treatment were minimal (less than 1 day), indicating that this is a preference-sensitive decision and shared decision making, is essential. Your patient indicates that he is keen to avoid developing active TB. He is not familiar with acute hepatitis; however, after you describe the condition to him, he is not particularly concerned. He has a positive IGRA test and goes on to receive latent TB treatment.
      a Tasillo et al.
      • Tasillo A.
      • Salomon J.A.
      • Trikalinos T.A.
      • Horsburgh Jr., C.R.
      • Marks S.M.
      • Linas B.P.
      Cost-effectiveness of testing and treatment for latent tuberculosis infection in residents born outside the United States with and without medical comorbidities in a simulation model.

      Are the Results Applicable to My Patient?

      You can address the applicability of the decision analysis to a specific patient by asking the following questions: 1) Is the population in the decision analysis similar to my patient? 2) Is the intervention (the intervention options) in the decision analysis similar to the intervention(s) under assessment in the clinical situation? 3) Is the main outcome of the decision analysis important to my patient for decision making?
      Discussing a patient’s preferences and values and ascertaining that they are consistent with those assumed in the model is essential. Focusing on elucidating patients’ preferences is particularly important if the study conclusion changed in the sensitivity analysis of utilities used in the decision analysis.

      Conclusion

      Decision analyses provide a systematic approach to assessing the risk-benefit balance of alternative strategies using objective evidence and careful, model-based evaluation. The information obtained from a decision analysis can be used for clinical decision making as well as decision making at a health policy level. When applying the results of a decision analysis, we first judge the credibility of the methods of the decision analysis and how confident we are in the preferred course of action.

      Supplemental Online Material

      References

        • Al-Rifai R.H.
        • Pearson F.
        • Critchley J.A.
        • Abu-Raddad L.J.
        Association between diabetes mellitus and active tuberculosis: a systematic review and meta-analysis.
        PLoS One. 2017; 12: e0187967
      1. Guidelines on the Management of Latent Tuberculosis Infection. World Health Organization, Geneva, Switzerland2015
        • Tasillo A.
        • Salomon J.A.
        • Trikalinos T.A.
        • Horsburgh Jr., C.R.
        • Marks S.M.
        • Linas B.P.
        Cost-effectiveness of testing and treatment for latent tuberculosis infection in residents born outside the United States with and without medical comorbidities in a simulation model.
        JAMA Intern Med. 2017; 177: 1755-1764
        • Tversky A.
        • Kahneman D.
        Judgment under uncertainty: heuristics and biases.
        Science. 1974; 185: 1124-1131
        • Caro J.J.
        • Briggs A.H.
        • Siebert U.
        • Kuntz K.M.
        • Force I-SMGRPT
        Modeling good research practices—overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—1.
        Value Health. 2012; 15: 796-803
        • Roberts M.
        • Russell L.B.
        • Paltiel A.D.
        • et al.
        Conceptualizing a model: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—2.
        Value Health. 2012; 15: 804-811
        • Siebert U.
        • Alagoz O.
        • Bayoumi A.M.
        • et al.
        State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—3.
        Value Health. 2012; 15: 812-820
        • Karnon J.
        • Stahl J.
        • Brennan A.
        • et al.
        Modeling using discrete event simulation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—4.
        Value Health. 2012; 15: 821-827
        • Pitman R.
        • Fisman D.
        • Zaric G.S.
        • et al.
        Dynamic transmission modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--5.
        Value Health. 2012; 15: 828-834
        • Briggs A.H.
        • Weinstein M.C.
        • Fenwick E.A.L.
        • et al.
        Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—6.
        Value Health. 2012; 15: 835-842
        • Eddy D.M.
        • Hollingworth W.
        • Caro J.J.
        • et al.
        Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—7.
        Value Health. 2012; 15: 843-850
        • Richardson W.S.
        • Detsky A.S.
        Users' guides to the medical literature. VII. How to use a clinical decision analysis. A. Are the results of the study valid? Evidence-Based Medicine Working Group.
        JAMA. 1995; 273: 1292-1295
        • Richardson W.S.
        • Detsky A.S.
        Users' guides to the medical literature. VII. How to use a clinical decision analysis. B. What are the results and will they help me in caring for my patients? Evidence Based Medicine Working Group.
        JAMA. 1995; 273: 1610-1613
        • Qaseem A.
        • Chou R.
        • Humphrey L.L.
        • Starkey M.
        • Shekelle P.
        Venous thromboembolism prophylaxis in hospitalized patients: a clinical practice guideline from the American College of Physicians.
        Ann Intern Med. 2011; 155: 625-632
        • Grant P.J.
        • Conlon A.
        • Chopra V.
        • Flanders S.A.
        Use of venous thromboembolism prophylaxis in hospitalized patients.
        JAMA Intern Med. 2018; 178: 1122-1124
        • Murad M.H.
        • Montori V.M.
        • Ioannidis J.P.
        • et al.
        How to read a systematic review and meta-analysis and apply the results to patient care: users' guides to the medical literature.
        JAMA. 2014; 312: 171-179
        • Diel R.
        • Nienhaus A.
        • Schaberg T.
        Cost-effectiveness of isoniazid chemoprevention in close contacts.
        Eur Respir J. 2005; 26: 465-473
        • Dobler C.C.
        • Martin A.
        • Marks G.B.
        Benefit of treatment of latent tuberculosis infection in individual patients.
        Eur Respir J. 2015; 46: 1397-1406
        • Taylor W.C.
        • Aronson M.D.
        • Delbanco T.L.
        Should young adults with a positive tuberculin test take isoniazid?.
        Ann Intern Med. 1981; 94: 808-813
        • Younis T.
        • Rayson D.
        • Skedgel C.
        The cost-utility of adjuvant chemotherapy using docetaxel and cyclophosphamide compared with doxorubicin and cyclophosphamide in breast cancer.
        Curr Oncol. 2011; 18: e288-e296
        • Swales J.D.
        Science in a health service.
        Lancet. 1997; 349: 1319-1321
        • Hunink M.
        • Glasziou P.
        • Siegel J.
        • et al.
        Decision Making in Health and Medicine: Integrating Evidence and Values.
        Cambridge University Press, New York, NY2001
        • Malhotra A.
        • Wu X.
        • Forman H.P.
        • Matouk C.C.
        • Gandhi D.
        • Sanelli P.
        Management of tiny unruptured intracranial aneurysms: a comparative effectiveness analysis.
        JAMA Neurol. 2018; 75: 27-34
        • Ryder H.F.
        • McDonough C.
        • Tosteson A.N.
        • Lurie J.D.
        Decision Analysis and Cost-effectiveness Analysis.
        Semin Spine Surg. 2009; 21: 216-222
        • Sanders G.D.
        • Neumann P.J.
        • Basu A.
        • et al.
        Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: Second Panel on Cost-Effectiveness in Health and Medicine.
        JAMA. 2016; 316: 1093-1103
      2. Neumann P.J. Sanders G.D. Russell L.B. Siegel J.E. Ganiats T.G. Cost-Effectiveness in Health and Medicine. 2nd Edition. Oxford University Press, New York, NY2016
        • Bang H.
        • Zhao H.
        Median-based incremental cost-effectiveness ratio (ICER).
        J Stat Theory Pract. 2012; 6: 428-442
        • Cohen D.J.
        • Reynolds M.R.
        Interpreting the results of cost-effectiveness studies.
        J Am Coll Cardiol. 2008; 52: 2119-2126
        • Dobler C.C.
        Screening strategies for active tuberculosis: focus on cost-effectiveness.
        Clinicoecon Outcomes Res. 2016; 8: 335-347
        • Sonnenberg F.A.
        • Beck J.R.
        Markov models in medical decision making: a practical guide.
        Med Decis Making. 1993; 13: 322-338
        • Chadha V.K.
        Tuberculosis epidemiology in India: a review.
        Int J Tuberc Lung Dis. 2005; 9: 1072-1082