Mayo Clinic Proceedings Home

Patient-Physician Agreement Using Summary Outcome Determination Scores

Published:December 05, 2017DOI:



      To determine whether the Summary Outcome Determination (SOD) score had exhibited a high level of physician-patient agreement in surgical patients.

      Patients and Methods

      The medical records of 320 postoperative patients were reviewed, of whom 164 patients were included in the study. Patients were included if both physician-assigned and patient-assigned SOD scores had been recorded. The SOD is administered as follows: the patient is asked “Compared to before surgery, is your elbow/shoulder better, worse or no different?” If better: “Is it improved, greatly improved, almost normal or normal?” If worse: “Is it worse or profoundly worse, or as bad as dying?” Each category is associated with a numerical value and definition for further clarification. The patient is asked to assign a category and a numerical value after the physician has already done so. These categories and values were evaluated between raters (ie, physician and patient) to assess reliability.


      The intraclass correlation coefficient of physician-patient numerical ratings was “excellent” (0.93). The Bland-Altman 95% limits of agreement on the differences between the physician and the patient ranged from −1.3 to 1.3. The physician and patient numerical rankings matched exactly in 118 patients (72%) or differed by a factor of no more than 1 (26%) in 161 (98%) patients.


      The SOD score can be used as both a surgeon-based and a patient-based outcome score, given the high level of agreement. Given its brevity, ease of understanding, and high interrater reliability, the SOD has the potential to be used across multiple specialties to rate outcomes.

      Abbreviations and Acronyms:

      ICC (intraclass correlation coefficient), SOD (Summary Outcome Determination)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Mayo Clinic Proceedings
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Ergina P.L.
        • Cook J.A.
        • Blazeby J.M.
        • et al.
        Challenges in evaluating surgical innovation.
        Lancet. 2009; 374: 1097-1104
        • Clancy C.M.
        • Eisenberg J.M.
        Outcomes research: measuring the end results of health care.
        Science. 1998; 282: 245-246
        • Amadio P.C.
        Outcomes measurements.
        J Bone Joint Surg Am. 1993; 75: 1583-1584
        • Johanson N.A.
        • Charlson M.E.
        • Szatrowski T.P.
        • Ranawat C.S.
        A self-administered hip-rating questionnaire for the assessment of outcome after total hip replacement.
        J Bone Joint Surg Am. 1992; 74: 587-597
        • Lieberman J.R.
        • Dorey F.
        • Shekelle P.
        • et al.
        Differences between patients' and physicians' evaluations of outcome after total hip arthroplasty.
        J Bone Joint Surg Am. 1996; 78: 835-838
        • Brokelman R.B.
        • van Loon C.J.
        • Rijnberg W.J.
        Patient versus surgeon satisfaction after total hip arthroplasty.
        J Bone Joint Surg Br. 2003; 85: 495-498
        • Roos E.M.
        Outcome after anterior cruciate ligament reconstruction—a comparison of patients' and surgeons' assessments.
        Scand J Med Sci Sports. 2001; 11: 287-291
        • Porter M.E.
        • Teisberg E.O.
        Redefining competition in health care.
        Harv Bus Rev. 2004; 82 (136): 64-76
        • Gandhi R.
        • Davey J.R.
        • Mahomed N.
        Patient expectations predict greater pain relief with joint arthroplasty.
        J Arthroplasty. 2009; 24: 716-721
        • Noble P.C.
        • Gordon M.J.
        • Weiss J.M.
        • Reddix R.N.
        • Conditt M.A.
        • Mathis K.B.
        Does total knee replacement restore normal knee function?.
        Clin Orthop Relat Res. 2005; 431: 157-165
        • Wylde V.
        • Blom A.W.
        Assessment of outcomes after hip arthroplasty.
        Hip Int. 2009; 19: 1-7
        • Bourne R.B.
        • Chesworth B.M.
        • Davis A.M.
        • Mahomed N.N.
        • Charron K.D.
        Patient satisfaction after total knee arthroplasty: who is satisfied and who is not?.
        Clin Orthop Relat Res. 2010; 468: 57-63
        • Bourne R.B.
        • Maloney W.J.
        • Wright J.G.
        An AOA critical issue: the outcome of the outcomes movement.
        J Bone Joint Surg Am. 2004; 86-A: 633-640
        • Höher J.
        • Bach T.
        • Münster A.
        • Bouillon B.
        • Tiling T.
        Does the mode of data collection change results in a subjective knee score? Self-administration versus interview.
        Am J Sports Med. 1997; 25: 642-647
        • Martin R.L.
        • Mohtadi N.G.
        • Safran M.R.
        • et al.
        Differences in physician and patient ratings of items used to assess hip disorders.
        Am J Sports Med. 2009; 37: 1508-1512
        • Gilbart M.K.
        • Gerber C.
        Comparison of the subjective shoulder value and the Constant score.
        J Shoulder Elbow Surg. 2007; 16: 717-721
        • Blonna D.
        • Lee G.C.
        • O'Driscoll S.W.
        Arthroscopic restoration of terminal elbow extension in high level athletes.
        Am J Sports Med. 2010; 38: 2509-2515
        • Haas M.
        Statistical methodology for reliability studies.
        J Manipulative Physiol Ther. 1991; 14: 119-132
        • Shrout P.E.
        • Fleiss J.L.
        Intraclass correlations: uses in assessing rater reliability.
        Psychol Bull. 1979; 86: 420-428
        • Cohen J.
        A coefficient of agreement for nominal scales.
        Educ Psychol Meas. 1960; 20: 37-46
        • Bland J.M.
        • Altman D.G.
        Statistical methods for assessing agreement between two methods of clinical measurement.
        Lancet. 1986; 1: 307-310
        • Bland J.M.
        • Altman D.G.
        Comparing methods of measurement: why plotting difference against standard method is misleading.
        Lancet. 1995; 346: 1085-1087
        • Dawson J.
        • Doll H.
        • Boller I.
        • et al.
        The development and validation of a patient-reported questionnaire to assess outcomes of elbow surgery.
        J Bone Joint Surg Br. 2008; 90: 466-473
        • Mintken P.E.
        • Glynn P.
        • Cleland J.A.
        Psychometric properties of the shortened disabilities of the Arm, Shoulder, and Hand Questionnaire (QuickDASH) and Numeric Pain Rating Scale in patients with shoulder pain.
        J Shoulder Elbow Surg. 2009; 18: 920-926
        • Sathyamoorthy P.
        • Kemp G.J.
        • Rawal A.
        • Rayner V.
        • Frostick S.P.
        Development and validation of an elbow score.
        Rheumatology (Oxford). 2004; 43 ([published correction appears in Rheumatology (Oxford). 2005;44(8):1081]): 1434-1440
        • Slover J.D.
        • Karia R.J.
        • Hauer C.
        • Gelber Z.
        • Band P.A.
        • Graham J.
        Feasibility of integrating standardized patient-reported outcomes in orthopedic care.
        Am J Manag Care. 2015; 21: e494-e500
        • Katz J.N.
        • Larson M.G.
        • Phillips C.B.
        • Fossel A.H.
        • Liang M.H.
        Comparative measurement sensitivity of short and longer health status instruments.
        Med Care. 1992; 30: 917-925
        • McGrory B.J.
        • Morrey B.F.
        • Rand J.A.
        • Ilstrup D.M.
        Correlation of patient questionnaire responses and physician history in grading clinical outcome following hip and knee arthroplasty.
        J Arthroplasty. 1996; 11: 47-57
        • Fitzpatrick R.
        • Fletcher A.
        • Gore S.
        • Jones D.
        • Spiegelhalter D.
        • Cox D.
        Quality of life measures in health care. I: Applications and issues in assessments.
        BMJ. 1992; 305: 1074-1077
        • Ghomrawi H.M.
        • Franco Ferrando N.
        • Mandl L.A.
        • Do H.
        • Noor N.
        • Gonzalez Della Valle A.
        How often are patient and surgeon recovery expectations for total joint arthroplasty aligned? Results of a pilot study.
        HSS J. 2011; 7: 229-234
        • O'Driscoll S.W.
        • Keeley F.W.
        • Salter R.B.
        The chondrogenic potential of free autogenous periosteal grafts for biological resurfacing of major full-thickness defects in joint surfaces under the influence of continuous passive motion: an experimental investigation in the rabbit.
        J Bone Joint Surg Am. 1986; 68: 1017-1035
        • O'Driscoll S.W.
        • Marx R.G.
        • Beaton D.E.
        • Miura Y.
        • Gallay S.H.
        • Fitzsimmons J.S.
        Validation of a simple histological-histochemical cartilage scoring system.
        Tissue Eng. 2001; 7: 313-320
        • Coe M.P.
        • Sutherland J.M.
        • Penner M.J.
        • Younger A.
        • Wing K.J.
        Minimal clinically important difference and the effect of clinical variables on the ankle osteoarthritis scale in surgically treated end-stage ankle arthritis.
        J Bone Joint Surg Am. 2015; 97: 818-823
        • Copay A.G.
        • Subach B.R.
        • Glassman S.D.
        • Polly Jr., D.W.
        • Schuler T.C.
        Understanding the minimum clinically important difference: a review of concepts and methods.
        Spine J. 2007; 7: 541-546
        • Liu S.
        • Diebo B.G.
        • Henry J.K.
        • et al.
        • International Spine Study Group (ISSG)
        The benefit of nonoperative treatment for adult spinal deformity: identifying predictors for reaching a minimal clinically important difference.
        Spine J. 2016; 16: 210-218
        • Malay S.
        • Chung K.C.
        • SUN Study Group
        The minimal clinically important difference after simple decompression for ulnar neuropathy at the elbow.
        J Hand Surg Am. 2013; 38: 652-659
        • Werner B.C.
        • Chang B.
        • Nguyen J.T.
        • Dines D.M.
        • Gulotta L.V.
        What change in American shoulder and elbow surgeons score represents a clinically important change after shoulder arthroplasty?.
        Clin Orthop Relat Res. 2016; 474 ([published correction appears in What Change in American Shoulder and Elbow Surgeons Score Represents a Clinically Important Change After Shoulder Arthroplasty? [Clin Orthop Relat Res. 2017]]): 2672-2681
        • Eichinger J.K.
        • Miller L.R.
        • Hartshorn T.
        • Li X.
        • Warner J.J.
        • Higgins L.D.
        Evaluation of satisfaction and durability after hemiarthroplasty and total shoulder arthroplasty in a cohort of patients aged 50 years or younger: an analysis of discordance of patient satisfaction and implant survival.
        J Shoulder Elbow Surg. 2016; 25: 772-780
        • Kaplan R.S.
        • Porter M.E.
        How to solve the cost crisis in health care.
        Harv Bus Rev. 2011; 89 (54, 56-61 passim): 46-52
        • Porter M.E.
        • Larsson S.
        • Lee T.H.
        Standardizing patient outcomes measurement.
        N Engl J Med. 2016; 374: 504-506
        • Herndon J.H.
        • Davidson S.M.
        • Apazidis A.
        Recent socioeconomic trends in orthopaedic practice.
        J Bone Joint Surg Am. 2001; 83-A: 1097-1105
        • Kleinman L.C.
        • Kosecoff J.
        • Dubois R.W.
        • Brook R.H.
        The medical appropriateness of tympanostomy tubes proposed for children younger than 16 years in the United States.
        JAMA. 1994; 271: 1250-1255
        • Winslow C.M.
        • Kosecoff J.B.
        • Chassin M.
        • Kanouse D.E.
        • Brook R.H.
        The appropriateness of performing coronary artery bypass surgery.
        JAMA. 1988; 260: 505-509