Advertisement
Mayo Clinic Proceedings Home

The Association Between Perceived Electronic Health Record Usability and Professional Burnout Among US Physicians

Open AccessPublished:November 14, 2019DOI:https://doi.org/10.1016/j.mayocp.2019.09.024

      Abstract

      Objective

      To describe and benchmark physician-perceived electronic health record (EHR) usability as defined by a standardized metric of technology usability and evaluate the association with professional burnout among physicians.

      Participants and Methods

      This cross-sectional survey of US physicians from all specialty disciplines was conducted between October 12, 2017, and March 15, 2018, using the American Medical Association Physician Masterfile. Among the 30,456 invited physicians, 5197 (17.1%) completed surveys. A random 25% (n=1250) of respondents in the primary survey received a subsurvey evaluating EHR usability, and 870 (69.6%) completed it. EHR usability was assessed using the System Usability Scale (SUS; range 0-100). SUS scores were normalized to percentile rankings across more than 1300 previous studies from other industries. Burnout was measured using the Maslach Burnout Inventory.

      Results

      Mean ± SD SUS score was 45.9±21.9. A score of 45.9 is in the bottom 9% of scores across previous studies and categorized in the “not acceptable” range or with a grade of F. On multivariate analysis adjusting for age, sex, medical specialty, practice setting, hours worked, and number of nights on call weekly, physician-rated EHR usability was independently associated with the odds of burnout with each 1 point more favorable SUS score associated with a 3% lower odds of burnout (odds ratio, 0.97; 95% CI, 0.97-0.98; P<.001).

      Conclusion

      The usability of current EHR systems received a grade of F by physician users when evaluated using a standardized metric of technology usability. A strong dose-response relationship between EHR usability and the odds of burnout was observed.

      Abbreviations and Acronyms:

      ATM (automated teller machine), DVR (digital video recorder), EHR (electronic health record), GPS (global positioning system), IQR (interquartile range), MBI (Maslach Burnout Inventory), OB/GYN (obstetrics and gynecology), SUS (System Usability Scale)
      The US health care system has undergone rapid computerization during the past decade. The 2009 Health Information Technology for Economic and Clinical Health Act provided $27 billion of federal incentives to encourage the adoption of electronic health records (EHRs) with the intent to improve health care quality, outcomes, and efficiency.
      • Blumenthal D.
      Stimulating the adoption of health information technology.
      These incentives stimulated large-scale adoption of EHRs in the United States, with 96% of nonfederal acute care hospitals and 86% of office-based physicians reporting EHR use by 2015 and 2017, respectively.
      • Henry J.
      • Pylypchuk Y.
      • Searcy T.
      • Patel V.
      Adoption of electronic health record systems among US non-federal acute care hospitals: 2008-2015.
      ,
      • Office of the National Coordinator for Health Information Technology
      Office-based physician electronic health record adoption.
      During this period of increasing computer use, physician professional dissatisfaction and burnout has also increased.
      • Shanafelt T.D.
      • Boone S.
      • Tan L.
      • et al.
      Burnout and satisfaction with work-life balance among US physicians relative to the general US population.
      • Friedberg M.W.
      • Chen P.G.
      • Van Busum K.R.
      • et al.
      Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy.
      • Shanafelt T.D.
      • Hasan O.
      • Dyrbye L.N.
      • et al.
      Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014.
      Physicians now spend 1 to 2 hours on EHRs and deskwork for every hour spent in direct face-to-face contact with patients, as well as an additional 1 to 2 hours of personal time on EHR-related activities daily outside of office hours.
      • Sinsky C.
      • Colligan L.
      • Li L.
      • et al.
      Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties.
      • Arndt B.G.
      • Beasley J.W.
      • Watkinson M.D.
      • et al.
      Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations.
      • Tai-Seale M.
      • Olson C.W.
      • Li J.
      • et al.
      Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine.
      Although time spent with the EHR has been attributed to the clerical burden of current documentation requirements,
      • Shanafelt T.D.
      • Dyrbye L.N.
      • Sinsky C.
      • et al.
      Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction.
      • Rassolian M.
      • Peterson L.E.
      • Fang B.
      • et al.
      Workplace factors associated with burnout of family physicians.
      • Downing N.L.
      • Bates D.W.
      • Longhurst C.A.
      Physician burnout in the electronic health record era: are we ignoring the real cause?.
      poor EHR usability has been found to be a contributor to physician dissatisfaction, and many have hypothesized a direct relationship between EHR usability and physician burnout.
      • Friedberg M.W.
      • Chen P.G.
      • Van Busum K.R.
      • et al.
      Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy.
      ,
      • Shanafelt T.D.
      • Dyrbye L.N.
      • Sinsky C.
      • et al.
      Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction.
      ,
      • American Medical Association
      Improving Care: Priorities to Improve Electronic Health Record Usability.
      • Babbott S.
      • Manwell L.B.
      • Brown R.
      • et al.
      Electronic medical records and physician stress in primary care: results from the MEMO Study.
      • Colligan L.
      • Sinsky C.
      • Schmidt-Bowman M.
      • Tutty M.
      Sources of physician satisfaction and dissatisfaction and review of administrative tasks in ambulatory practice: a qualitative analysis of physician and staff interviews. Published October 2016.
      • Olson K.
      • Sinsky C.
      • Rinne S.T.
      • et al.
      Cross-sectional survey of workplace stressors associated with physician burnout measured by the Mini-Z and the Maslach Burnout Inventory. Stress Health. 2019;35(2):157-175.
      Usability is “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.”
      • Carter J.
      • Harker S.
      ISO 9241-11 revised: what have we learnt about usability since 1998?.
      (p144) Having been used in more than 1300 studies, the System Usability Scale (SUS) is favored as an industry standard as a short, simple, and reliable measurement of technology usability with solid benchmarks to easily interpret its scores.
      • Brooke J.
      SUS-a quick and dirty usability scale.
      • Bangor A.
      • Kortum P.T.
      • Miller J.T.
      An empirical evaluation of the System Usability Scale.
      • Lewis J.R.
      • Sauro J.
      The factor structure of the System Usability Scale.
      Assistant Secretary for Public Affairs. System Usability Scale (SUS). September 2013.
      • Peres S.C.
      • Pham T.
      • Phillips R.
      Validation of the System Usability Scale (SUS). Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2013;57(1):192-196.
      • Brooke J.
      SUS: a retrospective. Journal of Usability Studies. 2013;8(2):29-40.
      For example, a Google search has a SUS score of 93, which ranks in approximately the top 0.01% of technologies evaluated using the SUS and can be described as being in the acceptable range with a usability grade of A. However, Microsoft Excel (Microsoft, 2009) has a SUS score of 57, which results in a ranking in the bottom 22% across studies with a low marginal acceptability and a usability grade of F (Figure 1).
      • Kortum P.T.
      • Bangor A.
      Usability ratings for everyday products measured with the System Usability Scale.
      • Sauro J.
      A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices.
      • Sauro J.
      • Lewis J.R.
      Quantifying the User Experience: Practical Statistics for User Research.
      Figure thumbnail gr1
      Figure 1System Usability Scale (SUS) score for the electronic health record (EHR) from the analysis reported here and compared across studies in other industries with everyday products mapped onto a grading scale, acceptability ranges, and percentile of scores. ATM = automated teller machine; DVR = digital video recorder; GPS = global positioning system.
      Figure adapted from: Kortum PT, Bangor A.
      • Kortum P.T.
      • Bangor A.
      Usability ratings for everyday products measured with the System Usability Scale.
      Usability ratings for everyday products measured with the System Usability Scale. International Journal of Human–Computer Interaction. 2013;29(2):67-76.—with permission from Taylor & Francis publishing, License Number 4594911446562.
      To our knowledge, few national-level data are available to assess the current state of EHR usability using a standardized metric or evaluate the relationship between usability and burnout. Although there has been much conjecture about the current level of EHR usability, it also has not been benchmarked relative to other technologies using a standardized metric of technology usability. To address these issues, we conducted a national survey of physician-perceived EHR usability using the SUS and evaluated the relationship between EHR usability and symptoms of professional burnout among US physicians. Given current EHR documentation time requirements and the qualitative association between EHR usability and physician dissatisfaction, we hypothesized that EHR SUS scores would be lower than industry standards for technology and correlate with increased rates of burnout as measured by using the Maslach Burnout Inventory (MBI).

      Participants and Methods

      As previously described,
      • Shanafelt T.D.
      • West C.P.
      • Sinsky C.
      • et al.
      Changes in burnout and satisfaction with work-life integration in physicians and the general US working population between 2011 and 2017.
      between October 12, 2017, and March 15, 2018, we surveyed a sample of US physicians from all specialty disciplines (with deliberate oversampling of non–primary care specialties) assembled using the American Medical Association Physician Masterfile. Among the 30,456 physicians who received an invitation to participate, 5197 (17.1%) completed surveys. A secondary survey with intensive follow-up was conducted in a sample of nonresponders to evaluate for response bias. Among the 476 individuals who did not respond to the electronic survey included in the secondary survey, 248 (52.1%) responded. Full details of the sampling strategy and the assessment for response bias have been previously reported, including analysis of the secondary survey results that indicated that participants were representative of US physicians.
      • Shanafelt T.D.
      • West C.P.
      • Sinsky C.
      • et al.
      Changes in burnout and satisfaction with work-life integration in physicians and the general US working population between 2011 and 2017.
      A random 25% (1250 of 5197) of responders completing the initial electronic survey completed a subsurvey evaluating their EHR’s usability. The Stanford University and Mayo Clinic institutional review boards reviewed and approved the study protocol.

      Study Measures

      Participants provided information for demographic characteristics (age, sex, and relationship status), medical specialty, hours worked per week, number of nights on call per week, practice setting, symptoms of burnout, and their perception of their EHR’s usability.

      EHR Usability

      Physician-perceived EHR usability was measured using the SUS. The SUS consists of a 10-item questionnaire on a 5-point Likert scale from Strongly Disagree to Strongly Agree.
      • Brooke J.
      SUS-a quick and dirty usability scale.
      Consistent with convention,
      • Bangor A.
      • Kortum P.T.
      • Miller J.T.
      An empirical evaluation of the System Usability Scale.
      ,
      • Lewis J.R.
      • Sauro J.
      The factor structure of the System Usability Scale.
      the language of the SUS was modified such that references to “the system” being evaluated were changed to “my EHR” (Supplement 1, available online at http://www.mayoclinicproceedings.org). The 10 items of the SUS are scored on a scale of 0 to 4, with each even-numbered question reverse coded. The items are summed and then multiplied by 2.5.
      Scores range from 0 to 100, for which higher scores indicate higher usability. Although the score ranges from 0 to 100, it is important to note that SUS scores are not percentiles.
      • Sauro J.
      A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices.
      Benchmarking scores to other products and studies often are helpful for interpreting an individual SUS score. The SUS scores follow a bell-shaped distribution across more than 1300 previous studies from other industries.
      Assistant Secretary for Public Affairs. System Usability Scale (SUS). September 2013.
      The scale can be normalized to a percentile ranking across previous studies using the SUS Calculator Package (MeasuringUsability.com, version 1.42; 2012). There are also acceptability ranges and grading scales to interpret SUS scores (Figure 1).
      • Kortum P.T.
      • Bangor A.
      Usability ratings for everyday products measured with the System Usability Scale.
      ,
      • Sauro J.
      A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices.
      For example, a SUS score of 68 is the average score across studies.

      Burnout

      Burnout was measured using the MBI, a validated questionnaire considered the criterion standard tool for measuring burnout.
      • Maslach C.
      • Jackson S.E.
      • Leiter M.P.
      Maslach Burnout Inventory Manual.
      • Rafferty J.P.
      • Lemkau J.P.
      • Purdy R.R.
      • Rudisill J.R.
      Validity of the Maslach Burnout Inventory for family practice physicians.
      • Lee R.T.
      • Ashforth B.E.
      A meta-analytic examination of the correlates of the three dimensions of job burnout.
      • Leiter M.P.
      • Durup J.
      The discriminant validity of burnout and depression: a confirmatory factor analytic study.
      Consistent with convention,
      • Shanafelt T.D.
      • Bradley K.A.
      • Wipf J.E.
      • Back A.L.
      Burnout and self-reported patient care in an internal medicine residency program.
      • Thomas N.K.
      Resident burnout.
      • Rosen I.M.
      • Gimotty P.A.
      • Shea J.A.
      • Bellini L.M.
      Evolution of sleep quantity, sleep deprivation, mood disturbances, empathy, and burnout among interns.
      individual physicians were considered to have professional burnout if they had a high score on the depersonalization (≥27) and/or emotional exhaustion subscales (≥10) of the MBI.
      • Maslach C.
      • Jackson S.E.
      • Leiter M.P.
      Maslach Burnout Inventory Manual.

      Statistical Analyses

      Standard descriptive statistics were used to characterize the physician sample that responded to the EHR usability subsurvey. Retired physicians were not included in the analysis.
      Associations between variables were evaluated using Kruskal-Wallis (continuous variables) or χ2 test (categorical variables), as appropriate. All tests were 2 sided, with a type I error level of 0.05. On preliminary analysis controlling for medical specialty, specialties with smaller numbers of participants in the subsurvey had considerable variability. To control for this variation, specialties with fewer than 20 participants were grouped in a pooled category of “Other” specialties. The Other category included these specialties (number of respondents in parentheses): neurosurgery (9), ophthalmology (8), otolaryngology (10), other (20), physical medicine and rehabilitation (15), preventive medicine and occupational medicine (4), radiation oncology (5), and urology (2). Multivariable analysis of differences in EHR usability and burnout was performed using linear and logistic regression, respectively. Demographic and professional factors (age, sex, hours worked per week, medical specialty, nights on call, and practice setting) were included in the models to identify characteristics associated with the dependent outcomes. The proportion of the variance in EHR usability and burnout that could be explained by medical specialty was determined using a mixed linear model (or multilevel approach) in which specialty was treated as a random effect. The proportion of the variance of burnout that could be explained by EHR usability was determined by evaluating the incremental R2 after adding SUS score to a multivariate linear regression model with MBI score as the dependent variable. All analyses were completed using R statistical software (version 3.5.3; R Foundation for Statistical Computing, Vienna, Austria).

      Results

      Of the 5197 responders in the master survey, a random 25% of responders (N=1250) received the EHR usability subsurvey. Among these individuals, the 870 (69.6%) who responded to all 10 SUS items were included in the analysis.
      The demographic characteristics of the 870 respondents to the subsurvey and the 5445 respondents to the full survey (5197 primary survey respondents + 248 secondary survey respondents) relative to all 890,083 practicing US physicians were generally similar (Table). Compared with physicians who responded to the secondary survey of nonresponders, the 870 participants in the EHR subsurvey were more likely to be women (40.6% [353 of 870] vs 30.4% [75 of 247]; P=.01); however, age (P=.7), years in practice (P=.7), and the prevalence of burnout (P=.5 (Supplement 2, available online at http://www.mayoclinicproceedings.org) were similar between the 2 groups. Collectively, these results suggest that participants in the EHR usability subsurvey appear generally representative of US physicians.
      TableDemographic Characteristics of Survey Responders and All US Physicians
      CharacteristicEHR Usability Subsurvey Responders (N=870)Full Survey Responders (N=5445)All US Physicians, 2017

      (N=890,083)
      Sex
       Male, n (%)509 (58.5)2907 (55.1)577,339 (64.9)
       Female, n (%)353 (40.6)1785 (33.8)311,776 (35.0)
       Other, n (%)1 (0.1)13 (0.2)NA
       Missing, n (%)7 (0.8)571 (10.8)968 (0.0)
      Age (y)
       Median (IQR)53 (42-61)53 (42-62)52
       <35, n (%)66 (7.4)305 (5.7)80,780 (9.1)
       35-44, n (%)206 (23.7)1117 (21.2)224,341 (25.2)
       45-54, n (%)184 (21.1)1095 (20.8)227,421 (25.6)
       55-64, n (%)264 (30.3)1343 (25.5)221,199 (24.9)
       ≥65, n (%)130 (14.9)799 (15.1)135,596 (15.2)
       Missing, n (%)22 (2.5)619 (11.7)746 (0.1)
      Primary care
       Yes222 (25.5)1243 (23.6)349,597 (39.3)
       No647 (74.4)3973 (75.3)540,486 (60.7)
       Missing1 (0.1)60 (1.1)
      Specialty
       Anesthesiology, n (%)38 (4.4)249 (4.7)
       Dermatology, n (%)23 (2.6)133 (2.5)
       Emergency medicine, n (%)54 (6.2)296 (5.6)
       Family medicine, n (%)55 (6.3)402 (7.6)
       Radiology, n (%)39 (4.5)221 (4.2)
       Neurology, n (%)33 (3.8)189 (3.6)
       Obstetrics and gynecology, n (%)42 (4.8)188 (3.6)
       Pathology, n (%)26 (3.0)145 (2.7)
       Psychiatry, n (%)54 (6.2)418 (7.9)
       Other, n (%)78 (9.0)617 (11.7)
       General internal medicine, n (%)82 (9.4)417 (7.9)
       Internal medicine subspecialty, n (%)113 (13)626 (11.9)
       General pediatrics, n (%)48 (5.5)254 (4.8)
       Pediatric subspecialty, n (%)53 (6.1)220 (4.2)
       General surgery, n (%)34 (3.9)155 (2.9)
       General surgery subspecialty, n (%)59 (6.8)392 (7.4)
       Orthopedic surgery, n (%)37 (4.3)270 (5.1)
       Missing, n (%)2 (0.2)84 (1.6)
      Hours worked per wk
       Median (IQR)50 (40-60)50 (40-60)
       <40 h, n (%)130 (15.8)860 (16.3)
       40-49 h, n (%)174 (20.0)1041 (19.7)
       50-59 h, n (%)222 (25.5)1230 (23.3)
       60-69 h, n (%)191 (22)1069 (20.3)
       70-79 h, n (%)79 (9.1)380 (7.2)
       ≥80 h, n (%)71 (8.2)363 (6.9)
       Missing, n (%)3 (0.3)333 (6.3)
      No. of nights on call per wk, median (IQR)1 (0-2)1 (0-2)
      Primary practice setting
       Private practice, n (%)393 (45.2)2474 (46.9)
       Academic medical center, n (%)280 (32.2)1394 (26.4)
       Veterans hospital, n (%)18 (2.1)107 (2.0)
       Active military practice, n (%)13 (1.5)55 (1.0)
       Other, n (%)165 (19.0)950 (18)
       Missing, n (%)1 (0.1)296 (5.6)
      Relationship status
       Single, n (%)106 (12.2)558 (10.6)
       Married, n (%)707 (81.3)3850 (73)
       Partnered, n (%)40 (4.6)196 (3.7)
       Widow/widower, n (%)8 (0.9)59 (1.1)
       Missing, n (%)9 (1.0)613 (11.6)
      EHR = electronic medical record; IQR = interquartile range; NA = not Available; SUS = System Usability Scale.
      Among the 870 physicians evaluating their EHR’s usability, mean ± SD SUS score was 45.9±21.9 (range, 0-100; interquartile range, 30-60). A score of 45.9 is in the bottom 9% of usability scores across studies in other industries and is categorized in the “not acceptable” range or with a grade of F. The distribution of scores generally followed a bell-shaped distribution with skewing to the lower end of the scale (Figure 2A). In aggregate, 733 of 870 (84.2%) respondents rated their EHR less than 68 on the SUS, the average score across industries.
      • Kortum P.T.
      • Bangor A.
      Usability ratings for everyday products measured with the System Usability Scale.
      ,
      • Sauro J.
      A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices.
      Figure thumbnail gr2
      Figure 2A, Distribution of electronic health record (EHR) System Usability Scale (SUS) scores across all subsurvey respondents. B, EHR SUS scores by medical specialty.
      Substantial variation in physician-rated EHR usability was observed by specialty (Figure 2B). Anesthesiology and general pediatrics had the highest raw SUS scores, whereas dermatology, orthopedic surgery, and general surgery had the lowest scores. On multivariable analysis adjusting for sex, hours worked, and nights on call, when compared with general internal medicine, practicing family medicine (−10.27; 95% CI, −17.73 to −2.81; P=.007), radiology (−9.08; 95% CI, −17.59 to −0.57; P=.040), other specialties (−7.58; 95% CI, −14.41 to −0.76; P=.030), general surgery (−13.93; 95% CI, −22.67 to −5.18; P=.002), and orthopedic surgery (−11.42; 95% CI, −19.85 to −2.98; P=.008) were all independently associated with lower SUS scores (Supplement 3, available online at http://www.mayoclinicproceedings.org). The proportion of the variance in EHR usability that can be explained by medical specialty was 2.0%.
      Additionally, SUS scores varied significantly by practice location (overall P=.003), with those who worked in an academic medical center rating their EHR less favorably (n=280; raw SUS score, 43.1±20; coefficient, −5.68; 95% CI, −9.22 to −2.13; P=.002) and those practicing in a veterans’ hospital rating their EHR more favorably (n=18; raw SUS score, 57.5±24.8; coefficient, 10.37; 95% CI, 0.13 to 20.61; P=.050) than physicians working in private practice (n=393; raw SUS score, 47.1±SD 23.3). Older physicians were more likely to rate their EHR as less usable (for each 1 year older, coefficient, −0.19; 95% CI, −0.32 to −0.05; P=.007). Although women were more likely to participate in the EHR usability subsurvey, no relationship was observed between sex and SUS scores on multivariable analysis (−0.17; 95% CI, −3.5 to 3.15; P=.9).
      Among the 865 of 870 (99.4%) EHR subsurvey responders who completed the MBI, mean ± SD emotional exhaustion score was 24.0±13.1 and mean ± SD depersonalization score was 7.1±6.7. Overall, 397 of 864 (45.9%) had at least 1 symptom of burnout (high emotional exhaustion and/or high depersonalization score). SUS scores were associated with emotional exhaustion, depersonalization, and overall burnout; as SUS scores increased, emotional exhaustion and depersonalization scores decreased, as did the overall prevalence of burnout (Figure 3A-C). Categorization of the 17 specialty discipline categories based on the specialty’s mean SUS score and prevalence of burnout is shown in Figure 4.
      Figure thumbnail gr3
      Figure 3Distribution of (A) emotional exhaustion scores, (B) depersonalization scores, and (C) percent respondents with burnout by System Usability Scale (SUS) score in 10-point categories.
      Figure thumbnail gr4
      Figure 4Electronic health record (EHR) usability as assessed by the System Usability Scale (SUS) and prevalence of burnout by specialty with regression line. OBGYN = obstetrics and gynecology.
      On multivariable analysis adjusting for sex, medical specialty, practice setting, hours worked, and number of nights on call, EHR SUS scores were independently associated with burnout. Each 1 point more favorable SUS score was associated with 3% lower odds of burnout (odds ratio, 0.97; 95% CI, 0.97-0.98; P<.001; Supplement 4, available online at http://www.mayoclinicproceedings.org). The proportion of variance in MBI scores that can be explained by SUS was 5.8%, and by medical specialty, was 3.3%.

      Discussion

      The findings of this national study of physicians indicate that the current state of EHR usability as measured using a standard technology usability scale is poor and considerably below the usability of many everyday technologies. On average, the EHR was given a usability rating far below that of Microsoft Excel, the lowest performing everyday item assessed by Kortum and Bangor
      • Kortum P.T.
      • Bangor A.
      Usability ratings for everyday products measured with the System Usability Scale.
      (Figure 1). Substantial variation in EHR usability ratings was observed by medical specialty and practice setting, which could reflect the variability of how EHRs are used to support clinical work and documentation across specialties.
      EHR usability scores were strongly and independently associated with physician burnout in a dose-response relationship. The odds of burnout were lower for each 1 point more favorable SUS score, a finding that persisted after adjusting for an extensive array of other personal and professional characteristics. The relationship between SUS score and burnout also persisted when emotional exhaustion and depersonalization were treated as continuous variables. Although the relationship between burnout and SUS score was strong, we are unable to determine causation or the potential direction of effect given the cross-sectional nature of the data. However, it is notable that EHR usability varied widely by specialty and that some of the specialties at higher risk for burnout rated their EHRs more favorably (eg, emergency medicine and neurology) than specialties associated with lower risk for burnout (radiology and surgery subspecialties). This finding suggests that the relationship between EHR usability and burnout may not be due to more burned out physicians rating their EHR less favorably. For example, emergency medicine physicians may benefit from the EHR’s ability to provide information rapidly yet still be at increased risk for burnout due to factors in the emergency department clinical context that are not EHR related.
      This analysis provides a global assessment of the current state of EHR usability across medical specialties and practice settings in the United States, as opposed to a specific vendor’s product. Because no 2 EHR installations are the same, even when supplied by the same vendor,
      • Richesson R.L.
      • Green B.B.
      • Laws R.
      • et al.
      Pragmatic (trial) informatics: a perspective from the NIH Health Care Systems Research Collaboratory.
      the findings provide general context and an objective measurement of the global usability gap between health care and other industries.
      Our study is subject to several limitations. As with all surveys, the potential for response bias and the representativeness of the sample are important questions. Our assessment for response bias was rigorous and included a comparison to both all US physicians and a random sample of nonresponders to the initial survey who participated in an incentivized secondary survey. This analysis indicated that participants were representative of US physicians with respect to age, years in practice, and prevalence of burnout. Although women were more likely to participate in the EHR usability subsurvey, no relationship between sex and SUS scores was observed.
      Second, relying only on physician perspectives on EHR usability does not account for perspectives of other EHR users such as patients, nurses, and other members of the care team.
      Third, although the SUS is favored as the industry standard for simple and reliable assessment of usability and allows comparison to the usability of other routinely used technologies, it has limitations. In particular, it is intended to assess: (1) satisfaction more so than the efficiency and effectiveness dimensions of usability and (2) specific tasks within a single system not a class of software like the EHR.
      Fourth, survey respondents may conflate their EHR usability with the burdens of documentation due to regulatory, clerical, or administrative requirements or local implementation that manifest in the EHR (eg, individual proficiency).
      • Downing N.L.
      • Bates D.W.
      • Longhurst C.A.
      Physician burnout in the electronic health record era: are we ignoring the real cause?.
      ,
      • Tutty M.A.
      • Carlasare L.E.
      • Lloyd S.
      • Sinsky C.A.
      The complex case of EHRs: examining the factors impacting the EHR user experience.
      ,
      • Longhurst C.A.
      • Davis T.
      • Maneker A.
      • et al.
      Arch Collaborative
      Local investment in training drives electronic health record user satisfaction.
      Specifically, institutional interpretations and implementation of state or federal regulations could manifest in profound differences in physician documentation requirements that the physician perceives as a deficiency in his or her EHR. For example, some hospitals require password revalidation before e-prescribing although the physician is already signed into the EHR, while a neighboring hospital may not.
      We are unaware of other national assessments of EHR usability across medical specialties and settings. The SUS has been used to measure EHR usability previously in a small study of 17 physicians assessing specific tasks in a controlled environment.
      • Clarke M.A.
      • Belden J.L.
      • Kim M.S.
      Determining differences in user performance between expert and novice primary care doctors when using an electronic health record (EHR).
      This study reported SUS scores of 68 to 70, considerably higher than the scores reported here that reflect a global assessment of EHR tasks as performed in real-world clinical contexts. The finding that the odds of burnout decrease with incrementally higher EHR usability is consistent with previous work suggesting that time spent with the EHR distracts from meaning in work, thus leading to burnout.
      • Gellert G.
      • Webster S.
      • Gillean J.
      • Melnick E.
      • Kanzaria H.
      Should US doctors embrace electronic health records?.
      ,
      • Flanagan M.E.
      • Militello L.G.
      • Rattray N.A.
      • Cottingham A.H.
      • Frankel R.M.
      The thrill is gone: burdensome electronic documentation takes its toll on physicians’ time and attention.
      A qualitative study identified EHR usability as a primary challenge to physician professional satisfaction.
      • Friedberg M.W.
      • Chen P.G.
      • Van Busum K.R.
      • et al.
      Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy.
      EHR stressors were found to be the most prevalent organizational stressors associated with burnout in a survey of 4118 clinicians across an academic health system.
      • Olson K.
      • Sinsky C.
      • Rinne S.T.
      • et al.
      Cross-sectional survey of workplace stressors associated with physician burnout measured by the Mini-Z and the Maslach Burnout Inventory. Stress Health. 2019;35(2):157-175.
      In a national physician survey conducted by members of our investigative team in 2014, physician use of EHR or computerized physician order entry was associated with lower satisfaction with the amount of time spent on clerical tasks.
      • Shanafelt T.D.
      • Dyrbye L.N.
      • Sinsky C.
      • et al.
      Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction.
      In the adjusted models controlling for age, sex, specialty, practice setting, and hours worked per week, computerized physician order entry was the aspect of EHR use most strongly related to burnout. An analysis of 9000 pediatric patient safety reports from 2012 to 2017 identified EHR usability issues contributing to a medication event in 36% (3243 of 9000) of reports, with 609 events resulting in patient harm.
      • Ratwani R.M.
      • Savage E.
      • Will A.
      • et al.
      Identifying electronic health record usability and safety challenges in pediatric settings.
      As noted, our study was intentionally designed to measure the overall state of EHR usability, not the usability of any specific vendor or instance of the EHR. The current variability in EHR usability across health care systems and vendors has been shown to be wide, with certain tasks having an average of a 9-fold difference in time and 8-fold difference in clicks between different EHRs.
      • Ratwani R.M.
      • Savage E.
      • Will A.
      • et al.
      A usability and safety analysis of electronic health records: a multi-center study.
      Therefore, the unacceptably low scores with wide variation reported here are likely representative of the current state of EHR usability; namely, that usability is generally poor but that there is a wide range of EHR usability across systems, specialties, and practice settings. In comparison to other technologies, current EHRs are still in a nascent form and are only going to become more complex moving forward.
      • Kharrazi H.
      • Gonzalez C.P.
      • Lowe K.B.
      • Huerta T.R.
      • Ford E.W.
      Forecasting the maturation of electronic health record functions among us hospitals: retrospective analysis and predictive model.
      Given this context, if EHR usability does not improve, increasing complexity could lead to compounded unintended effects on patient safety and physician burnout.
      Future research could explore the validity and reliability of using the SUS to measure EHR usability, as well as including assessment for other mediators or moderators (eg, work complexity moderator) in the potential causal relationship between EHR usability and physician burnout. In the meantime, establishing differences between SUS scores for the EHR and products from other industries could allow future temporal analyses of EHR usability to further characterize the EHR usability lifecycle and measure how system level changes that seek to improve EHR usability affect this usability gap.

      Conclusion

      The usability of current EHRs as assessed by US physicians using a standardized metric of technology usability is markedly lower than for most other technologies and falls into the grade category of F. A strong dose-response relationship between EHR usability and risk for burnout among physicians was observed. Given the association between EHR usability and physician burnout, improving EHR usability may be an important approach to help reduce health care professional burnout.

      Acknowledgments

      The authors thank Matthew Maleska for creating Figure 1.

      Supplemental Online Material

      Supplemental material can be found online at http://www.mayoclinicproceedings.org. Supplemental material attached to journal articles has not been edited, and the authors take responsibility for the accuracy of all data.

      References

        • Blumenthal D.
        Stimulating the adoption of health information technology.
        W V Med J. 2009; 105: 28-29
        • Henry J.
        • Pylypchuk Y.
        • Searcy T.
        • Patel V.
        Adoption of electronic health record systems among US non-federal acute care hospitals: 2008-2015.
        ONC Data Brief. 2016; 35: 1-9
        • Office of the National Coordinator for Health Information Technology
        Office-based physician electronic health record adoption.
        • Shanafelt T.D.
        • Boone S.
        • Tan L.
        • et al.
        Burnout and satisfaction with work-life balance among US physicians relative to the general US population.
        Arch Intern Med. 2012; 172: 1377-1385
        • Friedberg M.W.
        • Chen P.G.
        • Van Busum K.R.
        • et al.
        Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy.
        Rand Health Q. 2014; 3: 1
        • Shanafelt T.D.
        • Hasan O.
        • Dyrbye L.N.
        • et al.
        Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014.
        Mayo Clin Proc. 2015; 90: 1600-1613
        • Sinsky C.
        • Colligan L.
        • Li L.
        • et al.
        Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties.
        Ann Intern Med. 2016; 165: 753-760
        • Arndt B.G.
        • Beasley J.W.
        • Watkinson M.D.
        • et al.
        Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations.
        Ann Fam Med. 2017; 15: 419-426
        • Tai-Seale M.
        • Olson C.W.
        • Li J.
        • et al.
        Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine.
        Health Aff (Millwood). 2017; 36: 655-662
        • Shanafelt T.D.
        • Dyrbye L.N.
        • Sinsky C.
        • et al.
        Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction.
        Mayo Clin Proc. 2016; 91: 836-848
        • Rassolian M.
        • Peterson L.E.
        • Fang B.
        • et al.
        Workplace factors associated with burnout of family physicians.
        JAMA Intern Med. 2017; 177: 1036-1038
        • Downing N.L.
        • Bates D.W.
        • Longhurst C.A.
        Physician burnout in the electronic health record era: are we ignoring the real cause?.
        Ann Intern Med. 2018; 169: 50-51
        • American Medical Association
        Improving Care: Priorities to Improve Electronic Health Record Usability.
        American Medical Association, Chicago, IL2014
        • Babbott S.
        • Manwell L.B.
        • Brown R.
        • et al.
        Electronic medical records and physician stress in primary care: results from the MEMO Study.
        J Am Med Inform Assoc. 2014; 21: e100-e106
        • Colligan L.
        • Sinsky C.
        • Schmidt-Bowman M.
        • Tutty M.
        Sources of physician satisfaction and dissatisfaction and review of administrative tasks in ambulatory practice: a qualitative analysis of physician and staff interviews. Published October 2016.
        • Olson K.
        • Sinsky C.
        • Rinne S.T.
        • et al.
        Cross-sectional survey of workplace stressors associated with physician burnout measured by the Mini-Z and the Maslach Burnout Inventory. Stress Health. 2019;35(2):157-175.
        • Carter J.
        • Harker S.
        ISO 9241-11 revised: what have we learnt about usability since 1998?.
        In: Human-Computer Interaction: Design and Evaluation. Springer International Publishing, New York, NY2015: 143-151
        • Brooke J.
        SUS-a quick and dirty usability scale.
        in: Jordan P.W. Thomas B. McClelland I.L. Weerdmeester B. Usability Evaluation in Industry. Taylor & Francis Ltd, Bristol, PA1996: 189-194
        • Bangor A.
        • Kortum P.T.
        • Miller J.T.
        An empirical evaluation of the System Usability Scale.
        Int J Hum Comput Interact. 2008; 24: 574-594
        • Lewis J.R.
        • Sauro J.
        The factor structure of the System Usability Scale.
        in: Kurosu M. Human Centered Design. Vol 5619. Lecture Notes in Computer Science. Springer Berlin Heidelberg, Berlin, Germany2009: 94-103
      1. Assistant Secretary for Public Affairs. System Usability Scale (SUS). September 2013.
        • Peres S.C.
        • Pham T.
        • Phillips R.
        Validation of the System Usability Scale (SUS). Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2013;57(1):192-196.
        • Brooke J.
        SUS: a retrospective. Journal of Usability Studies. 2013;8(2):29-40.
        https://dl.acm.org/citation.cfm?id=2817913
        Date accessed: October 17, 2019
        • Kortum P.T.
        • Bangor A.
        Usability ratings for everyday products measured with the System Usability Scale.
        Int J Hum Comput Interact. 2013; 29: 67-76
        • Sauro J.
        A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices.
        Measuring Usability LLC, Denver, CO2011
        • Sauro J.
        • Lewis J.R.
        Quantifying the User Experience: Practical Statistics for User Research.
        2nd ed. Morgan Kaufmann, Burlington, MA2016
        • Shanafelt T.D.
        • West C.P.
        • Sinsky C.
        • et al.
        Changes in burnout and satisfaction with work-life integration in physicians and the general US working population between 2011 and 2017.
        Mayo Clin Proc. 2019; 94: 1681-1694
        • Maslach C.
        • Jackson S.E.
        • Leiter M.P.
        Maslach Burnout Inventory Manual.
        Consulting Psychologists Press, Palo Alto, CA1997
        • Rafferty J.P.
        • Lemkau J.P.
        • Purdy R.R.
        • Rudisill J.R.
        Validity of the Maslach Burnout Inventory for family practice physicians.
        J Clin Psychol. 1986; 42: 488-492
        • Lee R.T.
        • Ashforth B.E.
        A meta-analytic examination of the correlates of the three dimensions of job burnout.
        J Appl Psychol. 1996; 81: 123-133
        • Leiter M.P.
        • Durup J.
        The discriminant validity of burnout and depression: a confirmatory factor analytic study.
        Anxiety Stress Coping. 1994; 7: 357-373
        • Shanafelt T.D.
        • Bradley K.A.
        • Wipf J.E.
        • Back A.L.
        Burnout and self-reported patient care in an internal medicine residency program.
        Ann Intern Med. 2002; 136: 358-367
        • Thomas N.K.
        Resident burnout.
        JAMA. 2004; 292: 2880-2889
        • Rosen I.M.
        • Gimotty P.A.
        • Shea J.A.
        • Bellini L.M.
        Evolution of sleep quantity, sleep deprivation, mood disturbances, empathy, and burnout among interns.
        Acad Med. 2006; 81: 82-85
        • Richesson R.L.
        • Green B.B.
        • Laws R.
        • et al.
        Pragmatic (trial) informatics: a perspective from the NIH Health Care Systems Research Collaboratory.
        J Am Med Inform Assoc. 2017; 24: 996-1001
        • Tutty M.A.
        • Carlasare L.E.
        • Lloyd S.
        • Sinsky C.A.
        The complex case of EHRs: examining the factors impacting the EHR user experience.
        J Am Med Inform Assoc. 2019; 26: 673-677
        • Longhurst C.A.
        • Davis T.
        • Maneker A.
        • et al.
        • Arch Collaborative
        Local investment in training drives electronic health record user satisfaction.
        Appl Clin Inform. 2019; 10: 331-335
        • Clarke M.A.
        • Belden J.L.
        • Kim M.S.
        Determining differences in user performance between expert and novice primary care doctors when using an electronic health record (EHR).
        J Eval Clin Pract. 2014; 20: 1153-1161
        • Gellert G.
        • Webster S.
        • Gillean J.
        • Melnick E.
        • Kanzaria H.
        Should US doctors embrace electronic health records?.
        BMJ. 2017; 356: j242
        • Flanagan M.E.
        • Militello L.G.
        • Rattray N.A.
        • Cottingham A.H.
        • Frankel R.M.
        The thrill is gone: burdensome electronic documentation takes its toll on physicians’ time and attention.
        J Gen Intern Med. 2019; 34: 1096-1097
        • Ratwani R.M.
        • Savage E.
        • Will A.
        • et al.
        Identifying electronic health record usability and safety challenges in pediatric settings.
        Health Aff (Millwood). 2018; 37: 1752-1759
        • Ratwani R.M.
        • Savage E.
        • Will A.
        • et al.
        A usability and safety analysis of electronic health records: a multi-center study.
        J Am Med Inform Assoc. 2018; 25: 1197-1201
        • Kharrazi H.
        • Gonzalez C.P.
        • Lowe K.B.
        • Huerta T.R.
        • Ford E.W.
        Forecasting the maturation of electronic health record functions among us hospitals: retrospective analysis and predictive model.
        J Med Internet Res. 2018; 20: e10458

      Linked Article

      • In the Limelight: March 2020
        Mayo Clinic ProceedingsVol. 95Issue 3
        • Preview
          This monthly feature highlights three articles in the current print and online issue of Mayo Clinic Proceedings. These articles are also featured on the Mayo Clinic Proceedings’ YouTube Channel ( https://youtu.be/jZjkPg6qtTc ).
        • Full-Text
        • PDF