If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
To describe and benchmark physician-perceived electronic health record (EHR) usability as defined by a standardized metric of technology usability and evaluate the association with professional burnout among physicians.
Participants and Methods
This cross-sectional survey of US physicians from all specialty disciplines was conducted between October 12, 2017, and March 15, 2018, using the American Medical Association Physician Masterfile. Among the 30,456 invited physicians, 5197 (17.1%) completed surveys. A random 25% (n=1250) of respondents in the primary survey received a subsurvey evaluating EHR usability, and 870 (69.6%) completed it. EHR usability was assessed using the System Usability Scale (SUS; range 0-100). SUS scores were normalized to percentile rankings across more than 1300 previous studies from other industries. Burnout was measured using the Maslach Burnout Inventory.
Mean ± SD SUS score was 45.9±21.9. A score of 45.9 is in the bottom 9% of scores across previous studies and categorized in the “not acceptable” range or with a grade of F. On multivariate analysis adjusting for age, sex, medical specialty, practice setting, hours worked, and number of nights on call weekly, physician-rated EHR usability was independently associated with the odds of burnout with each 1 point more favorable SUS score associated with a 3% lower odds of burnout (odds ratio, 0.97; 95% CI, 0.97-0.98; P<.001).
The usability of current EHR systems received a grade of F by physician users when evaluated using a standardized metric of technology usability. A strong dose-response relationship between EHR usability and the odds of burnout was observed.
The US health care system has undergone rapid computerization during the past decade. The 2009 Health Information Technology for Economic and Clinical Health Act provided $27 billion of federal incentives to encourage the adoption of electronic health records (EHRs) with the intent to improve health care quality, outcomes, and efficiency.
These incentives stimulated large-scale adoption of EHRs in the United States, with 96% of nonfederal acute care hospitals and 86% of office-based physicians reporting EHR use by 2015 and 2017, respectively.
Physicians now spend 1 to 2 hours on EHRs and deskwork for every hour spent in direct face-to-face contact with patients, as well as an additional 1 to 2 hours of personal time on EHR-related activities daily outside of office hours.
(p144) Having been used in more than 1300 studies, the System Usability Scale (SUS) is favored as an industry standard as a short, simple, and reliable measurement of technology usability with solid benchmarks to easily interpret its scores.
For example, a Google search has a SUS score of 93, which ranks in approximately the top 0.01% of technologies evaluated using the SUS and can be described as being in the acceptable range with a usability grade of A. However, Microsoft Excel (Microsoft, 2009) has a SUS score of 57, which results in a ranking in the bottom 22% across studies with a low marginal acceptability and a usability grade of F (Figure 1).
To our knowledge, few national-level data are available to assess the current state of EHR usability using a standardized metric or evaluate the relationship between usability and burnout. Although there has been much conjecture about the current level of EHR usability, it also has not been benchmarked relative to other technologies using a standardized metric of technology usability. To address these issues, we conducted a national survey of physician-perceived EHR usability using the SUS and evaluated the relationship between EHR usability and symptoms of professional burnout among US physicians. Given current EHR documentation time requirements and the qualitative association between EHR usability and physician dissatisfaction, we hypothesized that EHR SUS scores would be lower than industry standards for technology and correlate with increased rates of burnout as measured by using the Maslach Burnout Inventory (MBI).
between October 12, 2017, and March 15, 2018, we surveyed a sample of US physicians from all specialty disciplines (with deliberate oversampling of non–primary care specialties) assembled using the American Medical Association Physician Masterfile. Among the 30,456 physicians who received an invitation to participate, 5197 (17.1%) completed surveys. A secondary survey with intensive follow-up was conducted in a sample of nonresponders to evaluate for response bias. Among the 476 individuals who did not respond to the electronic survey included in the secondary survey, 248 (52.1%) responded. Full details of the sampling strategy and the assessment for response bias have been previously reported, including analysis of the secondary survey results that indicated that participants were representative of US physicians.
A random 25% (1250 of 5197) of responders completing the initial electronic survey completed a subsurvey evaluating their EHR’s usability. The Stanford University and Mayo Clinic institutional review boards reviewed and approved the study protocol.
Participants provided information for demographic characteristics (age, sex, and relationship status), medical specialty, hours worked per week, number of nights on call per week, practice setting, symptoms of burnout, and their perception of their EHR’s usability.
Physician-perceived EHR usability was measured using the SUS. The SUS consists of a 10-item questionnaire on a 5-point Likert scale from Strongly Disagree to Strongly Agree.
the language of the SUS was modified such that references to “the system” being evaluated were changed to “my EHR” (Supplement 1, available online at http://www.mayoclinicproceedings.org). The 10 items of the SUS are scored on a scale of 0 to 4, with each even-numbered question reverse coded. The items are summed and then multiplied by 2.5.
Scores range from 0 to 100, for which higher scores indicate higher usability. Although the score ranges from 0 to 100, it is important to note that SUS scores are not percentiles.
Benchmarking scores to other products and studies often are helpful for interpreting an individual SUS score. The SUS scores follow a bell-shaped distribution across more than 1300 previous studies from other industries.
The scale can be normalized to a percentile ranking across previous studies using the SUS Calculator Package (MeasuringUsability.com, version 1.42; 2012). There are also acceptability ranges and grading scales to interpret SUS scores (Figure 1).
Standard descriptive statistics were used to characterize the physician sample that responded to the EHR usability subsurvey. Retired physicians were not included in the analysis.
Associations between variables were evaluated using Kruskal-Wallis (continuous variables) or χ2 test (categorical variables), as appropriate. All tests were 2 sided, with a type I error level of 0.05. On preliminary analysis controlling for medical specialty, specialties with smaller numbers of participants in the subsurvey had considerable variability. To control for this variation, specialties with fewer than 20 participants were grouped in a pooled category of “Other” specialties. The Other category included these specialties (number of respondents in parentheses): neurosurgery (9), ophthalmology (8), otolaryngology (10), other (20), physical medicine and rehabilitation (15), preventive medicine and occupational medicine (4), radiation oncology (5), and urology (2). Multivariable analysis of differences in EHR usability and burnout was performed using linear and logistic regression, respectively. Demographic and professional factors (age, sex, hours worked per week, medical specialty, nights on call, and practice setting) were included in the models to identify characteristics associated with the dependent outcomes. The proportion of the variance in EHR usability and burnout that could be explained by medical specialty was determined using a mixed linear model (or multilevel approach) in which specialty was treated as a random effect. The proportion of the variance of burnout that could be explained by EHR usability was determined by evaluating the incremental R2 after adding SUS score to a multivariate linear regression model with MBI score as the dependent variable. All analyses were completed using R statistical software (version 3.5.3; R Foundation for Statistical Computing, Vienna, Austria).
Of the 5197 responders in the master survey, a random 25% of responders (N=1250) received the EHR usability subsurvey. Among these individuals, the 870 (69.6%) who responded to all 10 SUS items were included in the analysis.
The demographic characteristics of the 870 respondents to the subsurvey and the 5445 respondents to the full survey (5197 primary survey respondents + 248 secondary survey respondents) relative to all 890,083 practicing US physicians were generally similar (Table). Compared with physicians who responded to the secondary survey of nonresponders, the 870 participants in the EHR subsurvey were more likely to be women (40.6% [353 of 870] vs 30.4% [75 of 247]; P=.01); however, age (P=.7), years in practice (P=.7), and the prevalence of burnout (P=.5 (Supplement 2, available online at http://www.mayoclinicproceedings.org) were similar between the 2 groups. Collectively, these results suggest that participants in the EHR usability subsurvey appear generally representative of US physicians.
TableDemographic Characteristics of Survey Responders and All US Physicians
EHR Usability Subsurvey Responders (N=870)
Full Survey Responders (N=5445)
All US Physicians, 2017 (N=890,083)
Male, n (%)
Female, n (%)
Other, n (%)
Missing, n (%)
<35, n (%)
35-44, n (%)
45-54, n (%)
55-64, n (%)
≥65, n (%)
Missing, n (%)
Anesthesiology, n (%)
Dermatology, n (%)
Emergency medicine, n (%)
Family medicine, n (%)
Radiology, n (%)
Neurology, n (%)
Obstetrics and gynecology, n (%)
Pathology, n (%)
Psychiatry, n (%)
Other, n (%)
General internal medicine, n (%)
Internal medicine subspecialty, n (%)
General pediatrics, n (%)
Pediatric subspecialty, n (%)
General surgery, n (%)
General surgery subspecialty, n (%)
Orthopedic surgery, n (%)
Missing, n (%)
Hours worked per wk
<40 h, n (%)
40-49 h, n (%)
50-59 h, n (%)
60-69 h, n (%)
70-79 h, n (%)
≥80 h, n (%)
Missing, n (%)
No. of nights on call per wk, median (IQR)
Primary practice setting
Private practice, n (%)
Academic medical center, n (%)
Veterans hospital, n (%)
Active military practice, n (%)
Other, n (%)
Missing, n (%)
Single, n (%)
Married, n (%)
Partnered, n (%)
Widow/widower, n (%)
Missing, n (%)
EHR = electronic medical record; IQR = interquartile range; NA = not Available; SUS = System Usability Scale.
Among the 870 physicians evaluating their EHR’s usability, mean ± SD SUS score was 45.9±21.9 (range, 0-100; interquartile range, 30-60). A score of 45.9 is in the bottom 9% of usability scores across studies in other industries and is categorized in the “not acceptable” range or with a grade of F. The distribution of scores generally followed a bell-shaped distribution with skewing to the lower end of the scale (Figure 2A). In aggregate, 733 of 870 (84.2%) respondents rated their EHR less than 68 on the SUS, the average score across industries.
Substantial variation in physician-rated EHR usability was observed by specialty (Figure 2B). Anesthesiology and general pediatrics had the highest raw SUS scores, whereas dermatology, orthopedic surgery, and general surgery had the lowest scores. On multivariable analysis adjusting for sex, hours worked, and nights on call, when compared with general internal medicine, practicing family medicine (−10.27; 95% CI, −17.73 to −2.81; P=.007), radiology (−9.08; 95% CI, −17.59 to −0.57; P=.040), other specialties (−7.58; 95% CI, −14.41 to −0.76; P=.030), general surgery (−13.93; 95% CI, −22.67 to −5.18; P=.002), and orthopedic surgery (−11.42; 95% CI, −19.85 to −2.98; P=.008) were all independently associated with lower SUS scores (Supplement 3, available online at http://www.mayoclinicproceedings.org). The proportion of the variance in EHR usability that can be explained by medical specialty was 2.0%.
Additionally, SUS scores varied significantly by practice location (overall P=.003), with those who worked in an academic medical center rating their EHR less favorably (n=280; raw SUS score, 43.1±20; coefficient, −5.68; 95% CI, −9.22 to −2.13; P=.002) and those practicing in a veterans’ hospital rating their EHR more favorably (n=18; raw SUS score, 57.5±24.8; coefficient, 10.37; 95% CI, 0.13 to 20.61; P=.050) than physicians working in private practice (n=393; raw SUS score, 47.1±SD 23.3). Older physicians were more likely to rate their EHR as less usable (for each 1 year older, coefficient, −0.19; 95% CI, −0.32 to −0.05; P=.007). Although women were more likely to participate in the EHR usability subsurvey, no relationship was observed between sex and SUS scores on multivariable analysis (−0.17; 95% CI, −3.5 to 3.15; P=.9).
Among the 865 of 870 (99.4%) EHR subsurvey responders who completed the MBI, mean ± SD emotional exhaustion score was 24.0±13.1 and mean ± SD depersonalization score was 7.1±6.7. Overall, 397 of 864 (45.9%) had at least 1 symptom of burnout (high emotional exhaustion and/or high depersonalization score). SUS scores were associated with emotional exhaustion, depersonalization, and overall burnout; as SUS scores increased, emotional exhaustion and depersonalization scores decreased, as did the overall prevalence of burnout (Figure 3A-C). Categorization of the 17 specialty discipline categories based on the specialty’s mean SUS score and prevalence of burnout is shown in Figure 4.
On multivariable analysis adjusting for sex, medical specialty, practice setting, hours worked, and number of nights on call, EHR SUS scores were independently associated with burnout. Each 1 point more favorable SUS score was associated with 3% lower odds of burnout (odds ratio, 0.97; 95% CI, 0.97-0.98; P<.001; Supplement 4, available online at http://www.mayoclinicproceedings.org). The proportion of variance in MBI scores that can be explained by SUS was 5.8%, and by medical specialty, was 3.3%.
The findings of this national study of physicians indicate that the current state of EHR usability as measured using a standard technology usability scale is poor and considerably below the usability of many everyday technologies. On average, the EHR was given a usability rating far below that of Microsoft Excel, the lowest performing everyday item assessed by Kortum and Bangor
(Figure 1). Substantial variation in EHR usability ratings was observed by medical specialty and practice setting, which could reflect the variability of how EHRs are used to support clinical work and documentation across specialties.
EHR usability scores were strongly and independently associated with physician burnout in a dose-response relationship. The odds of burnout were lower for each 1 point more favorable SUS score, a finding that persisted after adjusting for an extensive array of other personal and professional characteristics. The relationship between SUS score and burnout also persisted when emotional exhaustion and depersonalization were treated as continuous variables. Although the relationship between burnout and SUS score was strong, we are unable to determine causation or the potential direction of effect given the cross-sectional nature of the data. However, it is notable that EHR usability varied widely by specialty and that some of the specialties at higher risk for burnout rated their EHRs more favorably (eg, emergency medicine and neurology) than specialties associated with lower risk for burnout (radiology and surgery subspecialties). This finding suggests that the relationship between EHR usability and burnout may not be due to more burned out physicians rating their EHR less favorably. For example, emergency medicine physicians may benefit from the EHR’s ability to provide information rapidly yet still be at increased risk for burnout due to factors in the emergency department clinical context that are not EHR related.
This analysis provides a global assessment of the current state of EHR usability across medical specialties and practice settings in the United States, as opposed to a specific vendor’s product. Because no 2 EHR installations are the same, even when supplied by the same vendor,
the findings provide general context and an objective measurement of the global usability gap between health care and other industries.
Our study is subject to several limitations. As with all surveys, the potential for response bias and the representativeness of the sample are important questions. Our assessment for response bias was rigorous and included a comparison to both all US physicians and a random sample of nonresponders to the initial survey who participated in an incentivized secondary survey. This analysis indicated that participants were representative of US physicians with respect to age, years in practice, and prevalence of burnout. Although women were more likely to participate in the EHR usability subsurvey, no relationship between sex and SUS scores was observed.
Second, relying only on physician perspectives on EHR usability does not account for perspectives of other EHR users such as patients, nurses, and other members of the care team.
Third, although the SUS is favored as the industry standard for simple and reliable assessment of usability and allows comparison to the usability of other routinely used technologies, it has limitations. In particular, it is intended to assess: (1) satisfaction more so than the efficiency and effectiveness dimensions of usability and (2) specific tasks within a single system not a class of software like the EHR.
Fourth, survey respondents may conflate their EHR usability with the burdens of documentation due to regulatory, clerical, or administrative requirements or local implementation that manifest in the EHR (eg, individual proficiency).
Specifically, institutional interpretations and implementation of state or federal regulations could manifest in profound differences in physician documentation requirements that the physician perceives as a deficiency in his or her EHR. For example, some hospitals require password revalidation before e-prescribing although the physician is already signed into the EHR, while a neighboring hospital may not.
We are unaware of other national assessments of EHR usability across medical specialties and settings. The SUS has been used to measure EHR usability previously in a small study of 17 physicians assessing specific tasks in a controlled environment.
This study reported SUS scores of 68 to 70, considerably higher than the scores reported here that reflect a global assessment of EHR tasks as performed in real-world clinical contexts. The finding that the odds of burnout decrease with incrementally higher EHR usability is consistent with previous work suggesting that time spent with the EHR distracts from meaning in work, thus leading to burnout.
In a national physician survey conducted by members of our investigative team in 2014, physician use of EHR or computerized physician order entry was associated with lower satisfaction with the amount of time spent on clerical tasks.
In the adjusted models controlling for age, sex, specialty, practice setting, and hours worked per week, computerized physician order entry was the aspect of EHR use most strongly related to burnout. An analysis of 9000 pediatric patient safety reports from 2012 to 2017 identified EHR usability issues contributing to a medication event in 36% (3243 of 9000) of reports, with 609 events resulting in patient harm.
As noted, our study was intentionally designed to measure the overall state of EHR usability, not the usability of any specific vendor or instance of the EHR. The current variability in EHR usability across health care systems and vendors has been shown to be wide, with certain tasks having an average of a 9-fold difference in time and 8-fold difference in clicks between different EHRs.
Therefore, the unacceptably low scores with wide variation reported here are likely representative of the current state of EHR usability; namely, that usability is generally poor but that there is a wide range of EHR usability across systems, specialties, and practice settings. In comparison to other technologies, current EHRs are still in a nascent form and are only going to become more complex moving forward.
Given this context, if EHR usability does not improve, increasing complexity could lead to compounded unintended effects on patient safety and physician burnout.
Future research could explore the validity and reliability of using the SUS to measure EHR usability, as well as including assessment for other mediators or moderators (eg, work complexity moderator) in the potential causal relationship between EHR usability and physician burnout. In the meantime, establishing differences between SUS scores for the EHR and products from other industries could allow future temporal analyses of EHR usability to further characterize the EHR usability lifecycle and measure how system level changes that seek to improve EHR usability affect this usability gap.
The usability of current EHRs as assessed by US physicians using a standardized metric of technology usability is markedly lower than for most other technologies and falls into the grade category of F. A strong dose-response relationship between EHR usability and risk for burnout among physicians was observed. Given the association between EHR usability and physician burnout, improving EHR usability may be an important approach to help reduce health care professional burnout.
The authors thank Matthew Maleska for creating Figure 1.
Supplemental material can be found online at http://www.mayoclinicproceedings.org. Supplemental material attached to journal articles has not been edited, and the authors take responsibility for the accuracy of all data.
Stimulating the adoption of health information technology.
Grant Support: Funding for this study was provided by the Stanford Medicine WellMD Center, the American Medical Association, and the Mayo Clinic Department of Medicine Program on Physician Well-being. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Potential Competing Interests: The authors report no competing interests.
This monthly feature highlights three articles in the current print and online issue of Mayo Clinic Proceedings. These articles are also featured on the Mayo Clinic Proceedings’ YouTube Channel ( https://youtu.be/jZjkPg6qtTc ).