Advertisement
Mayo Clinic Proceedings Home

Clinician's Checklist for Reading and Using an Article About Patient-Reported Outcomes

      Abstract

      Clinicians need evidence-based medicine to help them make clinical decisions with their patients. For many health problems, the goal of treatment is to help the patient to function and feel better. To measure patient functioning, well-being, and symptoms, questionnaires referred to as patient-reported outcome (PRO) measures are often used. Clinicians are generally not trained in survey design, scale development, and questionnaire administration, making it difficult for them to interpret and effectively use PROs as clinical evidence. It is increasingly important that clinicians be able to understand and use outcomes measured from both the clinical and patient perspectives to inform their practice. We aim to provide a “Clinician's Checklist” to help practicing clinicians understand clinical research articles that include PROs so that the information can be used for decision making. This checklist provides an itemization of important areas for the reader to consider in evaluating research articles. We propose that clinicians consider 5 elements when reading a study using PROs: study design and PRO assessment strategy, PRO measure performance, validity of results, context of the findings, and generalizability to their own patient population. Patient-reported outcomes play an increasingly prominent role in clinical research and practice, and this trend has the potential to improve the patient-centeredness of care. Clinicians will need to understand how to use PROs to partner with patients and help them function and feel better. The proposed Clinician's Checklist can help clinicians systematically evaluate PRO studies by determining whether the study design was appropriate and whether the measurement approach was adequate and properly executed as well as by assisting in the interpretation and application of the results to a specific patient population.

      Abbreviations and Acronyms:

      FDA (Food and Drug Administration), ISOQOL (International Society for Quality of Life Research), PRO (patient-reported outcome)
      Patients go to clinicians for help with health problems or health maintenance. Some health problems are acute, such as newly developed shoulder pain, and some health problems are chronic, such as diabetes. Health maintenance focuses on preventive care and early detection. For many health problems today, the goal of treatment is to help the patient to function and feel better; measuring functioning and well-being is a critical aspect of patient-centered care.
      Institute of Medicine
      Crossing the Quality Chasm: A New Health System for the 21st Century.
      Patient-reported outcomes (PROs), which can be measured using questionnaires, assess the effect of care from the patient perspective.
      • Acquadro C.
      • Berzon R.
      • Dubois D.
      • et al.
      Incorporating the patient's perspective into drug development and communication: an ad hoc task force report of the Patient-Reported Outcomes (PRO) Harmonization Group meeting at the Food and Drug Administration, February 16, 2001.
      According to the Food and Drug Administration (FDA),A PRO is any report of the status of a patient's health condition that comes directly from the patient, without interpretation of the patient's response by a clinician or anyone else.
      Food and Drug Administration
      Guidance for Industry, Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims.
      Examples include symptom measures (eg, HIV Symptom Index
      • Justice A.C.
      • Holmes W.
      • Gifford A.L.
      • et al.
      Development and validation of a self-completed HIV symptom index.
      ), functional status (eg, activities of daily living scales
      • Katz S.
      Assessing self-maintenance: activities of daily living, mobility, and instrumental activities of daily living.
      ), and health-related quality of life
      • Guyatt G.H.
      • Feeny D.H.
      • Patrick D.L.
      Measuring health-related quality of life.
      (eg, the 36-item Short-Form Health Survey
      • Ware Jr., J.E.
      • Sherbourne C.D.
      The MOS 36-item short-form health survey (SF-36), I: conceptual framework and item selection.
      ). Patient-reported outcomes are increasingly included in randomized clinical trials, cohort studies, and comparative effectiveness research.
      • Wu A.W.
      • Snyder C.
      • Clancy C.M.
      • Steinwachs D.M.
      Adding the patient perspective to comparative effectiveness research.
      In some cases, PROs are decisive in demonstrating a superior therapy among several alternatives.
      • Blazeby J.M.
      • Avery K.
      • Sprangers M.
      • Pikhart H.
      • Fayers P.
      • Donovan J.
      Health-related quality of life measurement in randomized clinical trials in surgical oncology.
      In others, there may be discordance between clinical and PRO measures, such as when an antihypertensive medication lowers not only blood pressure but also health-related quality of life because of adverse effects.
      • Contopoulous-Loannidis D.G.
      • Karvouni A.
      • Kouri L.
      • Loannidis J.P.
      Reporting and interpretation of SF-36 outcomes in randomised trials: systematic review.
      • Wilson I.B.
      • Cleary P.D.
      Linking clinical variables with health-related quality of life: a conceptual model of patient outcomes.
      In either case, PROs can play an important role in informing treatment decisions.
      It is increasingly important that clinicians be able to understand and use outcomes measured from both the clinical and patient perspectives to inform their practice. However, there are various obstacles to practicing clinicians interested in using PRO results to inform patient care. Clinicians are generally not taught about the measurement and interpretation of PROs during their training, and so they have limited familiarity with these methods. As with other unfamiliar tests, clinicians may be skeptical or dismissive of PRO measures if they are uncertain about how to interpret and use them. The lack of education and training in this area makes it more difficult for clinicians to make use of PRO data from clinical studies in their practice.
      • Sprangers M.A.
      Disregarding clinical trial-based patient-reported outcomes is unwarranted: five advances to substantiate the scientific stringency of quality-of-life measurement.
      Other obstacles include the wide variety of different PRO measures used and how PRO findings are reported in the literature because this variation can lead to confusion and can impair clinicians' understanding of PROs.
      • Brundage M.
      • Bass B.
      • Davidson J.
      • et al.
      Patterns of reporting health-related quality of life outcomes in randomized clinical trials: implications for clinicians and quality of life researchers.
      In 1997, Guyatt et al
      • Guyatt G.H.
      • Naylor C.D.
      • Juniper E.
      • Heyland D.K.
      • Jaeschke R.
      • Cook D.J.
      Users' guides to the medical literature, XII: how to use articles about health-related quality of life. Evidence-Based Medicine Working Group.
      wrote a seminal article, intended for clinicians, on how to use studies about health-related quality of life. Since then, there have been several important changes in clinical research, practice, and PRO measurement methods. First, outcomes being considered have been broadened from health-related quality of life to include more explicit domains such as symptoms, functional impact, and patient satisfaction; this increase in PRO domains has led to an increase in the number of different questionnaires that might be considered. Second, the acceptance and application of evidence-based medicine is steadily increasing and clinicians are more likely to appreciate the importance of sound research methods. Admittedly, however, clinicians need to be educated about the scientific nature of PRO measurements. Third, an expanding pharmacopoeia increases the opportunities to measure PROs for the purpose of identifying effective treatments and for documenting symptoms and adverse effects associated with new drugs. Fourth, PRO measurement technology has advanced, including item response theory and computer adaptive testing, and the number of measurement tools has surged.
      • Cella D.F.
      • Tulsky D.S.
      • Gray G.
      • et al.
      The Functional Assessment of Cancer Therapy scale: development and validation of the general measure.
      Fifth, there has been a recent emphasis on integrating the patient perspective into comparative effectiveness research, with a focus on patient-centered outcomes research and patient-centered care.
      • Ahmed S.
      • Berzon R.A.
      • Revicki D.A.
      • et al.
      International Society for Quality of Life Research
      The use of patient-reported outcomes (PRO) within comparative effectiveness research: implications for clinical practice and health care policy.
      This has been underlined by health care legislation recently passed in the United States that established the new Patient-Centered Outcomes Research Institute.

      111th US Congress. The Patient Protection and Affordable Care Act (ACA). Pub L No. 111-148. March 23, 2010.

      Finally, PROs are beginning to be incorporated into electronic health records, increasing the opportunity for clinicians to use them.
      • Wu A.W.
      • Snyder C.
      • Clancy C.M.
      • Steinwachs D.M.
      Adding the patient perspective to comparative effectiveness research.
      Thus, it is increasingly important that clinicians be able to evaluate the PRO methods used and the PRO findings reported in the published literature.
      The purpose of the “Clinician's Checklist” is to help practicing clinicians apply in their patient care the results of clinical research articles that include PROs. Building on the foundation of Guyatt et al's work,
      • Guyatt G.H.
      • Naylor C.D.
      • Juniper E.
      • Heyland D.K.
      • Jaeschke R.
      • Cook D.J.
      Users' guides to the medical literature, XII: how to use articles about health-related quality of life. Evidence-Based Medicine Working Group.
      we offer an approach and a brief checklist to help clinicians review research studies that include PROs. We also provide “working examples” of how such an article can be evaluated and used.

      The Proposed Checklist

      We propose that 5 elements are most important to consider when reading a published study using PROs: assessment strategy and the study design, the performance of the PRO tool, the validity of the results, the context of the results, and generalizability to one's own patient population. These are based in part on the guidelines published by Guyatt et al.
      • Guyatt G.H.
      • Naylor C.D.
      • Juniper E.
      • Heyland D.K.
      • Jaeschke R.
      • Cook D.J.
      Users' guides to the medical literature, XII: how to use articles about health-related quality of life. Evidence-Based Medicine Working Group.
      Following a general description of each element, we formulate the questions in a “Clinician's Checklist” to help clinicians judge a study's adequacy (Table 1).
      Table 1Clinician's Checklist to Evaluate Studies Using PROs
      1. Was the PRO assessment strategy appropriate?
       a. PRO hypothesis stated?A priori hypothesis explicit for PROs
       b. PRO measures described?PRO measures used, and timing/follow-up of subjects
       c. PRO content appropriate?Investigators measured aspects of patients' lives that patients consider important
      PRO domains correspond to anticipated effects of disease and treatment
      All important aspects of patient-reported outcomes included
      2. Did they measure PRO effectively?
       a. Evidence for reliability, validity?The PRO instruments appear to work as intended: evidence of internal consistency and/or test retest reliability, and construct validity are cited or are well established
       b. Were missing data handled appropriately?Similar number of questionnaires completed by respondents in all treatment groups at every time point

      Missing data management strategy described

      Presence of data analysis plan for handling death, if frequent
      3. Should I believe the results?
       a. Internal validityFindings established; observed effects likely to be caused by intervention
      If nontreatment factors affect PRO, risk adjustment used
      4. Were the results placed in clinical context?
       a. Was clinical meaning of results explained?Magnitude of effect on PROs described

      Clinical importance of observed differences in PRO scores demonstrated
       b. Will the results help me in caring for my patients?Benefits and harms recognized and reconciled, including potential trade-offs between quality and quantity of life

      Description of what a clinician should do with the results; study information helps clinician communicate with patients about treatment options; applicability of group results to an individual patient.
      5. Do the results apply to my patients?
       a. External validity to clinician's practiceStudy population is similar enough to clinician's patient population to apply to practice
      PRO = patient-reported outcome.

      Was the PRO Assessment Strategy Appropriate?

      Elements that are important to the conceptualization and design of any clinical research study apply equally to studies including PROs. The research question, study design, patient population of interest, and primary outcomes should be identified. Justification should be provided for the PROs that are selected. Is there existing evidence that suggests an intervention or treatment has an effect on some aspect of PROs? Does the PRO measure assess that effect? Ideally, there should have been relevant patient input in the development and testing of the PRO; otherwise, the PRO may have been developed by clinicians to reflect what clinicians think is important, rather than what is important to patients. The article should indicate the primary and secondary outcomes and whether these are measured from the clinical, patient, or societal perspective. Patient-reported outcome hypotheses should be stated explicitly a priori.
      The PRO measurement strategy should be described, including the timing of initial and follow-up assessments; this timing should be consistent with knowledge about the expected trajectory of patient outcomes over time in the disease setting of interest and, if possible, based on any information regarding the timing of treatment-related changes in patient status. It is critical that a study assess pretreatment quality of life or symptom severity and that the follow-up be long enough to assess differences demanded by the hypothesis. The PRO content should correspond to the extent and breadth of problems observed in the patient population.
      To evaluate these characteristics, the reader should determine first whether the basic study design was sound and then whether the measurement strategy would allow the study to capture the effects of treatment on patient outcomes. Although there is often pressure to measure only symptoms and adverse effects, it is important to evaluate the “reach” of these symptoms to the patient's day-to-day functioning. For example, a phase II trial may have a more restricted focus on symptoms but a phase III study should have a more comprehensive assessment of the effect of treatment on patient functioning. The reader should check to see whether important aspects of patient-reported outcomes have been omitted, because their omission could lead to incorrect conclusions.
      • Contopoulous-Loannidis D.G.
      • Karvouni A.
      • Kouri L.
      • Loannidis J.P.
      Reporting and interpretation of SF-36 outcomes in randomised trials: systematic review.

      Was the PRO Measured Effectively?

      In examining a research article, the reader should determine whether there is sufficient evidence cited to suggest that the PRO measures should “work” when used to test the study hypotheses. Effective measurement of the patient's health status is important to drawing meaningful conclusions. A distribution of PRO scores should be observed in the study population to ensure the study's ability to discriminate between groups or identify changes related to the intervention. The Methods section should cite evidence of the PRO measure's internal consistency reliability, test-retest reliability, and construct validity in the clinical population.
      • Haywood K.L.
      • Garratt A.M.
      • Fitzpatrick R.
      Quality of life in older people: a structured review of generic self-assessed health instruments.
      • Reeve B.B.
      • Wyrwich K.W.
      • Wu A.W.
      • et al.
      ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research.
      There should be evidence that the questionnaire is “responsive,” for example, that changes in scores parallel demonstrable changes in patient health over time. In addition, because missing PRO data are not likely to be missing at random, the authors should outline a plan for handling missing data. If a substantial incidence of death is anticipated, the method of handling this should be planned for in the analysis. The results should describe the extent and pattern of missing data. The absence of any aforementioned elements should lead the reader to question the study findings, particularly if the conclusions suggest no treatment effect or no difference between groups.

      Are the Results Believable?

      The PRO results should be clearly described. The study's internal validity should be established, addressing whether the observed effects likely result from the intervention. To do so, the authors should establish the comparability of treatment groups at baseline and ensure that known confounding variables have been measured. When nontreatment factors are known to affect PRO scores, a system for risk adjustment should be applied to ensure fair comparison between groups. Results should be presented for important patient subgroups that might be expected to show heterogeneity of treatment effects.
      • Weiss C.O.
      • Segal J.B.
      • Varadhan R.
      Assessing the applicability of trial evidence to a target sample in the presence of heterogeneity of treatment effect.
      Ideally, these subgroups should be identified a priori or results should be qualified as exploratory.
      To evaluate the internal validity of a study, the reader should assess whether it seems likely that the observed results can be attributed to the intervention rather than to other factors, whether a risk adjustment strategy was used successfully, and finally, whether they believe the effects are clinically plausible.

      Are Results Placed in Clinical Context?

      The clinical significance of PRO results must be discussed explicitly, including whether the change was large enough to be noticeable to the patient or to compel a treatment change. Patient-reported outcomes provide more comprehensive information about positive and negative effects of disease and treatments and the “reach” of symptoms and adverse effects to day-to-day patient functioning. If an intervention has both positive and negative effects, the discussion should balance benefits and harms.
      EuroQol Group
      EuroQol–a new facility for the measurement of health-related quality of life.
      This is important when there are trade-offs between quality and quantity of life, such as when a treatment extends life but decreases quality of life (eg, toxic chemotherapy). Preference-based measures of health-related quality of life, such as the standard gamble and the time trade-off, and rating scales (EuroQol five-dimensional questionnaire
      • Kaplan R.M.
      • Anderson J.P.
      • Wu A.W.
      • Mathews W.C.
      • Kozin F.
      • Orenstein D.
      The Quality of Well-being Scale: applications in AIDS, cystic fibrosis, and arthritis.
      and the Quality of Well-being Scale
      • Torrance G.W.
      Utility approach to measuring health-related quality of life.
      ) can be useful in these situations because they can integrate effects on morbidity and mortality into a single score. It is also important when there is significant mortality in the study, leaving the patient to choose which factor trumps the others. Given a study's PRO results, it may or may not be obvious what management option a clinician would consider, but including recommendations from the authors increases the likelihood that the study findings will be translated to practice change.
      The reader should identify the magnitude of effect on the patient's quality of life and determine whether it is large enough to motivate changes in patient care. The reader should consider potential trade-offs involving the benefits and harms suggested by the study findings.

      Does This Apply to My Patients?

      External validity of the findings is important to clinicians if they are going to engage in a dialogue with patients about treatment options. The reader should judge how well the study simulates clinical practice in general, and whether or not the results are generalizable to his or her own patient population. Ideally, study authors will address the generalizability of study results, including PROs,
      • Brundage M.
      • Bass B.
      • Davidson J.
      • et al.
      Patterns of reporting health-related quality of life outcomes in randomized clinical trials: implications for clinicians and quality of life researchers.
      to help clinicians with this task.

      The PRO Clinician's Checklist: Application to 2 Studies

      For illustrative purposes, we apply the checklist to 2 published articles to illustrate its use when evaluating clinical studies and determining their usefulness for one's own practice (Tables 2 and 3). The first example
      • Neumayer L.
      • Giobbie-Hurder A.
      • Jonasson O.
      • et al.
      Open mesh versus laparoscopic mesh repair of inguinal hernia.
      evaluates a study comparing 2 surgical techniques to repair inguinal hernia. The primary outcome of interest was hernia recurrence; the secondary outcomes—pain and functional status—were measured using PRO instruments, a visual analog scale for pain
      Agency for Health Care Policy and Research
      Acute Pain Management: Operative or Medical Procedures and Trauma (Clinical Practice Guideline).
      and the 36-item Short-Form Health Survey.
      • Ware Jr., J.E.
      • Sherbourne C.D.
      The MOS 36-item short-form health survey (SF-36), I: conceptual framework and item selection.
      These seemed to have appropriate content, evidence for reliability and validity, and timing of administration. Missing data were not mentioned and the clinical meaning of scores was not explained, but the results seemed plausible. The study found that although open surgery was associated with a lower hernia recurrence, laparoscopic repair resulted in less pain and faster return to better functional status. However, it should be noted that if missing data were substantial and informative, the results for both arms could be biased because of the inclusion of healthier patients better able to respond to treatment.
      Table 2Evaluation of Study
      • Neumayer L.
      • Giobbie-Hurder A.
      • Jonasson O.
      • et al.
      Open mesh versus laparoscopic mesh repair of inguinal hernia.
      Using PRO Clinician's Checklist
      1. Was the PRO assessment strategy appropriate?
       a. PRO hypothesis stated?PRO hypotheses not stated explicitly; Implication: Laparoscopic repair would result in less pain and faster return to better functional status than open repair.
       b. PRO measures described?Pain: VAS
      Agency for Health Care Policy and Research
      Acute Pain Management: Operative or Medical Procedures and Trauma (Clinical Practice Guideline).
      ; Functional status or activity levels: SF-36
      • Ware Jr., J.E.
      • Sherbourne C.D.
      The MOS 36-item short-form health survey (SF-36), I: conceptual framework and item selection.
      PRO administration timing somewhat arbitrary from the perspective of accepted surgical recovery; however, time points seem appropriate. Because one of the rationales for laparoscopic hernia repair is faster return to normal function and less pain, early measurements appropriate. Timing matched to hypothesized time points and relevant to the intervention: (1) baseline, (2) completion of intervention, (3) 6-mo follow-up.
       c. PRO content appropriate?Instruments measured pain, functional status, and activity levels; did not necessarily measure all aspects that patients consider important, eg, sleep, sexual function.
      2. Did they measure PRO effectively?
       a. Evidence for reliability, validity?Reliability for both VAS
      Agency for Health Care Policy and Research
      Acute Pain Management: Operative or Medical Procedures and Trauma (Clinical Practice Guideline).
      and SF-36
      • Ware Jr., J.E.
      • Sherbourne C.D.
      The MOS 36-item short-form health survey (SF-36), I: conceptual framework and item selection.
      widely studied in various populations, including those similar to the study population. However, reliability and validity of instruments not reported in study.
      Both instruments well validated for pain measurement and functional status. VAS valid and reliable for postoperative pain assessment. Neither specifically assessed for the measurement of pain or functional status in inguinal hernia or its operative repair.
       b. Were missing data handled appropriately?No report on percentages of missing data at any point in the study; no explanation of how missing data were handled. Handling of death was not specified.
      3. Should I believe the results?
       a. Internal validityIt is unclear how the dropout rate affected the statistical analysis; several comparisons were not statistically significant, and a beta error was not determined.
      Multivariate analyses used to adjust for stratification variables (study site and treatment at baseline), age, and baseline value of the outcome.
      4. Were the results placed in clinical context?
       a. Was clinical meaning of results explained?Magnitude of effect described; no mention of the clinical meaning of scores provided.
      Benefits of the intervention discussed; report no adverse events, aside from more pain noted in the “open” group.
       b. Will the results help me in caring for my patients?No economic evaluation of trade-offs identified between quality and quantity of life.

      Authors suggest how the reader can use the information to improve decision making with patients.
      5. Do the results apply to my patients?
       a. External validity to clinician's practiceOnly male patients included in this study, with higher frequency of African American patients than the general population. These factors should be considered when assessing relevance to own patient population.
      PRO = patient-reported outcome; SF-36 = 36-item Short-Form Health Survey; VAS = visual analog scale.
      Table 3Evaluation of Study Number 2
      • Mutrie N.
      • Campbell A.M.
      • Whyte F.
      • et al.
      Benefits of supervised group exercise programme for women being treated for early stage breast cancer: progmatic randomised controlled trial.
      Using PRO Clinician's Checklist
      1. Was the PRO assessment strategy appropriate?
       a. PRO hypothesis stated?Hypothesis stated: “12 weeks of supervised group exercise … would improve quality of life for women during treatment for early stage breast cancer … benefits maintained for six months after the intervention.” Patients randomized to intervention (group exercise) or usual treatment (no group exercise) groups.
       b. PRO measures described?FACT-General
      • Cella D.F.
      • Tulsky D.S.
      • Gray G.
      • et al.
      The Functional Assessment of Cancer Therapy scale: development and validation of the general measure.
      presented as primary outcome measure, with the 4 subscales described. Secondary PRO measures: BDI
      • Beck A.
      • Steer R.
      • Brown G.
      Manual for the Beck Depression Inventory-2.
      and PANAS.
      • Watson D.
      • Clark L.A.
      • Tellegen A.
      Development and validation of brief measures of positive and negative affect: the PANAS scales.
      Other secondary measures of physical activity: body mass index, 12-minute walk test, shoulder mobility test. References provided for FACT scales' and other PRO measures' development and validation; measures described in general terms as “appropriate for use with cancer patients.”
      Data collected (baseline, the end of the intervention, 6 mo) appropriate given the intervention and the hypothesis.
       c. PRO content appropriate?Outcomes assessed are supported by the brief literature review presented; appropriate for the hypothesis.
      2. Did they measure PRO effectively?
       a. Evidence for reliability, validity?No data presented on the reliability and validity of the PRO data from this study; reliability and validity of these questionnaires previously established.
       b. Were missing data handled appropriately?A flowchart of allocation and assessment provided. The intervention group had greater loss to follow-up than did the control group. Deaths were rare events in both groups.
      No specific discussion of analytic approaches to address missing data.
      3. Should I believe the results?
       a. Internal validityThe authors acknowledge that the primary hypothesis was not supported. Presentation of the results focuses on outcome differences found between the groups assigned to exercise and usual care. Significant differences (P<.0001) and trends identified (eg, 12-min walk and shoulder mobility) are difficult to interpret given the failure to meet the primary end point and the lack of specification regarding which particular PRO domains were expected to differ between groups.
      4. Were the results placed in clinical context?
       a. Was clinical meaning of results explained? Will the results help me in caring for my patients?The authors note that it is difficult to determine what part of the exercise program was associated with the benefits and that participation in the group itself may have been valuable. The findings on the shoulder and walk tests support physical benefits.
      5. Do the results apply to my patients?
       a. External validity to clinician's practicePrevious research established the potential of exercise to improve physical and psychosocial aspects of quality of life among breast cancer survivors during and after treatment. Although interpretation is difficult without statistically significant results on the primary end point, the authors promote the various positive findings from the study. This study used a group-based exercise program; generalizability therefore limited to settings in which group exercise could be implemented. “Many” participants could not attend the classes because of the required commuting time to class.
      BDI = Beck Depression Inventory; FACT = Functional Assessment of Cancer Therapy; PANAS = Positive And Negative Affect Scale; PRO = patient-reported outcome.
      Absent a universally agreed best outcome, explicit consideration of various functional and psychological outcomes can help clarify patients' values and facilitate a decision. This study concludes with a recommendation for the open-surgery approach based on the primary outcome of recurrence but that individual patients may assess the risks and benefits differently. Faced with the evidence from this article, some might choose the laparoscopic approach with its significantly reduced postoperative pain and quicker return to normal activities. For an individual with limited sick leave, this information might be decisive.
      The second evaluation examines a randomized controlled trial
      • Mutrie N.
      • Campbell A.M.
      • Whyte F.
      • et al.
      Benefits of supervised group exercise programme for women being treated for early stage breast cancer: progmatic randomised controlled trial.
      of a group exercise program for women with early-stage breast cancer. Both primary and secondary outcomes—quality of life, depression, and affect—were measured using PROs: the Functional Assessment of Cancer Therapy – General,
      • Cella D.F.
      • Tulsky D.S.
      • Gray G.
      • et al.
      The Functional Assessment of Cancer Therapy scale: development and validation of the general measure.
      the Beck Depression Inventory,
      • Beck A.
      • Steer R.
      • Brown G.
      Manual for the Beck Depression Inventory-2.
      and the Positive and Negative Affect Scale.
      • Watson D.
      • Clark L.A.
      • Tellegen A.
      Development and validation of brief measures of positive and negative affect: the PANAS scales.
      The design and the conduct of the study satisfactorily supported the findings, suggesting no effect on the overall functioning and well-being of the women enrolled in this trial. Because some findings based on tests of shoulder functioning and walking suggested benefits of exercise, the study recommended supervised group exercise. However, the authors also acknowledged that the requirement for subjects to travel to classes may have diminished interest in the intervention.

      Discussion

      The quality of PRO research studies affects the usefulness of the resulting PRO data for decision making. As end users of these data, clinicians face several challenges to making effective use of research articles with PROs. First, inadequacies in study design and execution may preclude drawing conclusions. The study may overlook an important quality-of-life issue. The measures may not perform well. The study sample may not be generalizable to many real-world populations. Second, the presentation of the methods and results may handicap the clinician, such as by providing limited information about the study population. Third, several practical issues for using PRO assessment are not entirely settled. Although substantial progress has been made in our ability to translate the meaning of scores for clinicians, it can still be difficult to interpret their meaning, regarding both differences between groups and changes over time. The challenge we face is in applying guidelines for interpreting clinically meaningful changes in an individual's PRO score and devising treatment plans on the basis of that score when our accumulated data from PRO scores guiding our interpretations is based on group averages from clinical trials or observational studies. An individual score is subject to more measurement error than aggregate scores from groups of patients.
      • Donaldson G.
      Patient-reported outcomes and the mandate of measurement.
      Despite these occasional shortcomings, articles with PROs can help the clinician and patient make decisions about which course of action will likely result in the best outcome. The PRO Clinician's Checklist proposed in this article can help clinicians systematically evaluate such studies, helping them to determine whether the study was conducted with sufficient rigor for the results to be believed and to interpret and apply the results to a specific patient population. The intention is that the checklist helps clinicians to evaluate the relevance of scientific studies that use PROs for their own patients and practice. If the results from a study are believable, if the PRO end points are salient to one's patient population, and the magnitude of the results can be understood, then the clinician will be in a stronger position to integrate PRO results with other clinical data and individual patient preferences. This should lead to improved decision making, greater patient satisfaction, and improved outcomes.
      The classic article published by Guyatt et al
      • Guyatt G.H.
      • Naylor C.D.
      • Juniper E.
      • Heyland D.K.
      • Jaeschke R.
      • Cook D.J.
      Users' guides to the medical literature, XII: how to use articles about health-related quality of life. Evidence-Based Medicine Working Group.
      was aimed squarely at the practicing clinician, posing 3 questions to consider when reading an article about health-related quality of life: “Are the results valid?” “What were the results?” and “Will the results help me in caring for my patients?” These placed relatively little emphasis on study design and measurement methods. Since then, clinicians have begun to factor quality-of-life considerations into their decision making with patients more regularly. Regulators, led by the FDA, and scientific bodies emphasize the importance of applying standardized and rigorous methods to studies including PROs, beginning with the framing of a priori hypotheses about effects on PROs.
      Food and Drug Administration
      Guidance for Industry, Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims.
      Accordingly, our article emphasizes some of the methodological aspects of clinical research using PROs; these studies should (1) provide an explicit statement of hypotheses, (2) document patient input in the development and testing of the PROs used in the research, (3) describe methods for avoiding bias in the analysis of data (particularly in the context of substantial missing data), and (4) explain the clinical meaning of PRO findings.
      Several factors portend a more prominent role for PROs in clinical research in the upcoming decade. First, the typical adult of the future will be more than 50 years old living with multiple chronic conditions.
      • Anderson G.
      Medicare and chronic conditions.
      For these individuals, patient-centered outcomes research identifies the best strategies to attain optimum levels of functioning and well-being. Thus, it is likely that there will be an increase in the number of published research studies using PROs. There has also been increased attention to PROs in systematic reviews.
      • Feeny D.H.
      • Eckstrom E.N.
      • Whitlock E.P.
      • Perdue L.A.
      A Primer for Systematic Reviewers and Others on the Measurement of Functional Status and Health-Related Quality of Life in Older Adults.
      • Johnston B.C.
      • Patrick D.L.
      • Busse J.W.
      • Schnunemann H.J.
      • Agarwal A.
      • Guyatt G.H.
      Patient-reported outcomes in meta-analyses–part 1: assessing risk of bias and combining outcomes.
      Second, there have been advances and increased standardization in the measurement of PROs, particularly using both qualitative and quantitative methods.
      • Cella D.
      • Riley W.
      • Stone A.
      • et al.
      PROMIS Cooperative Group
      The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008.
      • Liu H.
      • Cella D.
      • Gershon R.
      • et al.
      Representativeness of the Patient-Reported Outcomes Measurement Information System Internet panel.
      • Rothrock N.E.
      • Hays R.D.
      • Spritzer K.
      • Yount S.E.
      • Riley W.
      • Cella D.
      Relative to the general US population, chronic diseases are associated with poorer health-related quality of life as measured by the Patient-Reported Outcomes Measurement Information System (PROMIS).
      The Patient-Reported Outcomes Measurement Information System initiative is an example of a national effort to standardize PRO measures (eg, pain, fatigue, physical function, depression, and anxiety) across diseases and health conditions
      • Cella D.
      • Riley W.
      • Stone A.
      • et al.
      PROMIS Cooperative Group
      The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008.
      and to have measures that are appropriate to use and interpret across disease conditions. Third, more and more PRO data are being presented to clinicians, with the digitization of medical records and the development of patient portals with the capacity to collect patient-reported data.
      • Wu A.W.
      • Snyder C.
      • Clancy C.M.
      • Steinwachs D.M.
      Adding the patient perspective to comparative effectiveness research.
      Much needs to be accomplished to increase the use of PRO evidence in clinical decision making. First, researchers and funders must insist on higher quality reporting of PRO studies to strengthen the evidence base. Important recent efforts include the development of reporting standards for PROs in randomized clinical trials and a project to expand the Consolidated Standards of Reporting Trials to include PROs.
      • Calvert M.
      • Blazeby J.
      • Altman D.G.
      • Revicki D.A.
      • Moher D.
      • Brundage M.D.
      CONSORT PRO Group
      Reporting of patient reported outcomes in randomized trials: the CONSORT PRO extension.
      • Brundage M.
      • Blazeby J.
      • Revicki D.
      • et al.
      Patient-reported outcomes in randomized clinical trials: development of ISOQOL reporting standards.
      Several resources are now available to help the researcher conduct competent PRO studies, such as the FDA document released in 2009.
      • Mokkink L.B.
      • Terwee C.B.
      • Patrick D.L.
      • et al.
      The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study.
      Second, a consensus definition of terms is necessary, such as that compiled by the Consensus-based Standards for the selection of health Measurement Instruments initiative.
      • Mokkink L.B.
      • Terwee C.B.
      • Patrick D.L.
      • et al.
      The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study.
      Third, policymakers must continue to support the inclusion of PROs in clinical trials and other forms of research. Finally, links should be sought between traditional sources of data for health care research and PROs.
      • Wu A.W.
      • Snyder C.
      • Clancy C.M.
      • Steinwachs D.M.
      Adding the patient perspective to comparative effectiveness research.
      Providing information on treatment effects from both the clinical and patient perspective should ultimately enhance the decision-making process.

      Conclusion

      Clinicians need education and training on how to incorporate information about PROs into their practice and how to apply it to treatment decisions for individual patients. Tools such as the Clinician's Checklist may help make studies that use PROs more accessible to those with the greatest opportunity to use them.

      Acknowledgments

      This article was reviewed by members of the International Society for Quality of Life Research (ISOQOL) and is endorsed by the ISOQOL Board of Directors. The tools and mechanisms described in this article are not being endorsed by the ISOQOL. The instruments and projects presented are examples and are not an exhaustive list. We also acknowledge 2 anonymous reviewers who suggested useful improvements to the article.

      References

        • Institute of Medicine
        Crossing the Quality Chasm: A New Health System for the 21st Century.
        Committee on Quality of Health Care in America, Institute of Medicine, Washington, DC2001
        • Acquadro C.
        • Berzon R.
        • Dubois D.
        • et al.
        Incorporating the patient's perspective into drug development and communication: an ad hoc task force report of the Patient-Reported Outcomes (PRO) Harmonization Group meeting at the Food and Drug Administration, February 16, 2001.
        Value Health. 2003; 6: 522-531
        • Food and Drug Administration
        Guidance for Industry, Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims.
        Office of Communications, Division of Drug Information Center for Drug Evaluation and Research Food and Drug Administration, Silver Spring, MD2009
        • Justice A.C.
        • Holmes W.
        • Gifford A.L.
        • et al.
        Development and validation of a self-completed HIV symptom index.
        J Clin Epidemiol. 2001; 54: S77-S90
        • Katz S.
        Assessing self-maintenance: activities of daily living, mobility, and instrumental activities of daily living.
        J Am Geriatric Soc. 1983; 31: 721-727
        • Guyatt G.H.
        • Feeny D.H.
        • Patrick D.L.
        Measuring health-related quality of life.
        Ann Intern Med. 1993; 118: 622-629
        • Ware Jr., J.E.
        • Sherbourne C.D.
        The MOS 36-item short-form health survey (SF-36), I: conceptual framework and item selection.
        Med Care. 1992; 30: 473-483
        • Wu A.W.
        • Snyder C.
        • Clancy C.M.
        • Steinwachs D.M.
        Adding the patient perspective to comparative effectiveness research.
        Health Aff (Millwood). 2010; 29: 1863-1871
        • Blazeby J.M.
        • Avery K.
        • Sprangers M.
        • Pikhart H.
        • Fayers P.
        • Donovan J.
        Health-related quality of life measurement in randomized clinical trials in surgical oncology.
        J Clin Oncol. 2006; 24: 3178-3186
        • Contopoulous-Loannidis D.G.
        • Karvouni A.
        • Kouri L.
        • Loannidis J.P.
        Reporting and interpretation of SF-36 outcomes in randomised trials: systematic review.
        BMJ. 2009; 338: a3006
        • Wilson I.B.
        • Cleary P.D.
        Linking clinical variables with health-related quality of life: a conceptual model of patient outcomes.
        JAMA. 1995; 273: 59-65
        • Sprangers M.A.
        Disregarding clinical trial-based patient-reported outcomes is unwarranted: five advances to substantiate the scientific stringency of quality-of-life measurement.
        Acta Oncol. 2010; 49: 155-163
        • Brundage M.
        • Bass B.
        • Davidson J.
        • et al.
        Patterns of reporting health-related quality of life outcomes in randomized clinical trials: implications for clinicians and quality of life researchers.
        Qual Life Res. 2011; 20: 653-664
        • Guyatt G.H.
        • Naylor C.D.
        • Juniper E.
        • Heyland D.K.
        • Jaeschke R.
        • Cook D.J.
        Users' guides to the medical literature, XII: how to use articles about health-related quality of life. Evidence-Based Medicine Working Group.
        JAMA. 1997; 277: 1232-1237
        • Cella D.F.
        • Tulsky D.S.
        • Gray G.
        • et al.
        The Functional Assessment of Cancer Therapy scale: development and validation of the general measure.
        J Clin Oncol. 1993; 11: 570-579
        • Ahmed S.
        • Berzon R.A.
        • Revicki D.A.
        • et al.
        • International Society for Quality of Life Research
        The use of patient-reported outcomes (PRO) within comparative effectiveness research: implications for clinical practice and health care policy.
        Med Care. 2012; 50: 1060-1070
      1. 111th US Congress. The Patient Protection and Affordable Care Act (ACA). Pub L No. 111-148. March 23, 2010.

        • Haywood K.L.
        • Garratt A.M.
        • Fitzpatrick R.
        Quality of life in older people: a structured review of generic self-assessed health instruments.
        Qual Life Res. 2005; 14: 1651-1668
        • Reeve B.B.
        • Wyrwich K.W.
        • Wu A.W.
        • et al.
        ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research.
        Qual Life Res. 2013; 22: 1889-1905
        • Weiss C.O.
        • Segal J.B.
        • Varadhan R.
        Assessing the applicability of trial evidence to a target sample in the presence of heterogeneity of treatment effect.
        Pharmacoepidemiol Drug Saf. 2012; 21: 121-129
        • EuroQol Group
        EuroQol–a new facility for the measurement of health-related quality of life.
        Health Policy. 1990; 16: 199-208
        • Kaplan R.M.
        • Anderson J.P.
        • Wu A.W.
        • Mathews W.C.
        • Kozin F.
        • Orenstein D.
        The Quality of Well-being Scale: applications in AIDS, cystic fibrosis, and arthritis.
        Med Care. 1989; 27: S27-S43
        • Torrance G.W.
        Utility approach to measuring health-related quality of life.
        J Chronic Dis. 1987; 40: 593-603
        • Neumayer L.
        • Giobbie-Hurder A.
        • Jonasson O.
        • et al.
        Open mesh versus laparoscopic mesh repair of inguinal hernia.
        N Engl J Med. 2004; 350: 1819-1827
        • Agency for Health Care Policy and Research
        Acute Pain Management: Operative or Medical Procedures and Trauma (Clinical Practice Guideline).
        Acute Pain Management Guideline Panel, Bethesda, MD1992
        • Mutrie N.
        • Campbell A.M.
        • Whyte F.
        • et al.
        Benefits of supervised group exercise programme for women being treated for early stage breast cancer: progmatic randomised controlled trial.
        BMJ. 2007; 334: 517
        • Beck A.
        • Steer R.
        • Brown G.
        Manual for the Beck Depression Inventory-2.
        Psychological Corporation, San Antonio, TX1996
        • Watson D.
        • Clark L.A.
        • Tellegen A.
        Development and validation of brief measures of positive and negative affect: the PANAS scales.
        J Pers Soc Psychol. 1988; 54: 1063-1070
        • Donaldson G.
        Patient-reported outcomes and the mandate of measurement.
        Qual Life Res. 2008; 17: 1303-1313
        • Anderson G.
        Medicare and chronic conditions.
        N Engl J Med. 2005; 353: 305-309
        • Feeny D.H.
        • Eckstrom E.N.
        • Whitlock E.P.
        • Perdue L.A.
        A Primer for Systematic Reviewers and Others on the Measurement of Functional Status and Health-Related Quality of Life in Older Adults.
        Agency for Healthcare Research and Quality, 2013
        • Johnston B.C.
        • Patrick D.L.
        • Busse J.W.
        • Schnunemann H.J.
        • Agarwal A.
        • Guyatt G.H.
        Patient-reported outcomes in meta-analyses–part 1: assessing risk of bias and combining outcomes.
        Health Qual Life Outcomes. 2013; 11: 109
        • Cella D.
        • Riley W.
        • Stone A.
        • et al.
        • PROMIS Cooperative Group
        The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008.
        J Clin Epidemiol. 2010; 63: 1179-1194
        • Liu H.
        • Cella D.
        • Gershon R.
        • et al.
        Representativeness of the Patient-Reported Outcomes Measurement Information System Internet panel.
        J Clin Epidemiol. 2010; 63: 1169-1178
        • Rothrock N.E.
        • Hays R.D.
        • Spritzer K.
        • Yount S.E.
        • Riley W.
        • Cella D.
        Relative to the general US population, chronic diseases are associated with poorer health-related quality of life as measured by the Patient-Reported Outcomes Measurement Information System (PROMIS).
        J Clin Epidemiol. 2010; 63: 1195-1204
        • Calvert M.
        • Blazeby J.
        • Altman D.G.
        • Revicki D.A.
        • Moher D.
        • Brundage M.D.
        • CONSORT PRO Group
        Reporting of patient reported outcomes in randomized trials: the CONSORT PRO extension.
        JAMA. 2013; 309: 814-822
        • Brundage M.
        • Blazeby J.
        • Revicki D.
        • et al.
        Patient-reported outcomes in randomized clinical trials: development of ISOQOL reporting standards.
        Qual Life Res. 2013; 22: 1161-1175
        • Mokkink L.B.
        • Terwee C.B.
        • Patrick D.L.
        • et al.
        The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study.
        Qual Life Res. 2010; 19: 539-549