Advertisement
Mayo Clinic Proceedings Home

An Electronic Health Record–Compatible Model to Predict Personalized Treatment Effects From the Diabetes Prevention Program: A Cross-Evidence Synthesis Approach Using Clinical Trial and Real-World Data

Open AccessPublished:November 13, 2021DOI:https://doi.org/10.1016/j.mayocp.2021.09.012

      Abstract

      Objective

      To develop an electronic health record (EHR)-based risk tool that provides point-of-care estimates of diabetes risk to support targeting interventions to patients most likely to benefit.

      Patients and Methods

      A risk prediction model was developed and validated in a large observational database of patients with an index visit date between January 1, 2012, and December 31, 2016, with treatment effect estimates from risk-based reanalysis of clinical trial data. The risk model development cohort included 1.1 million patients with prediabetes from the OptumLabs Data Warehouse (OLDW); the validation cohort included a distinct sample of 1.1 million patients in OLDW. The randomly assigned clinical trial cohort included 3081 people from the Diabetes Prevention Program (DPP) study.

      Results

      Eleven variables reliably obtainable from the EHR were used to predict diabetes risk. This model validated well in the OLDW (C statistic = 0.76; observed 3-year diabetes rate was 1.8% (95% confidence interval [CI], 1.7 to 1.9) in the lowest-risk quarter and 19.6% (19.4 to 19.8) in the highest-risk quarter). In the DPP, the hazard ratio (HR) for lifestyle modification was constant across all levels of risk (HR, 0.43; 95% CI, 0.35 to 0.53), whereas the HR for metformin was highly risk dependent (HR, 1.1; 95% CI, 0.61 to 2.0 in the lowest-risk quarter vs HR, 0.45; 95% CI, 0.35 to 0.59 in the highest-risk quarter). Fifty-three percent of the benefits of population-wide dissemination of the DPP lifestyle modification and 73% of the benefits of population-wide metformin therapy can be obtained by targeting the highest-risk quarter of patients.

      Conclusion

      The Tufts–Predictive Analytics and Comparative Effectiveness DPP Risk model is an EHR-compatible tool that might support targeted diabetes prevention to more efficiently realize the benefits of the DPP interventions.

      Abbreviations and Acronyms:

      AA (Black/African American), ADA (American Diabetes Association), BMI (body mass index), DPP (Diabetes Prevention Program), EHR (electronic health record), FG (fasting glucose), HbA1c (hemoglobin A1c), HDL (high-density lipoprotein), HR (hazard ratio), HTN (hypertension), IRB (institutional review board), OLDW (OptumLabs Data Warehouse)
      The Diabetes Prevention Program (DPP) Study showed that either an intensive program of lifestyle modification or pharmacotherapy with metformin substantially reduced the risk for developing type 2 diabetes in patients at high risk, compared with “usual care.”
      • Knowler W.C.
      • Barrett-Connor E.
      • Fowler S.E.
      • et al.
      Diabetes Prevention Program Research Group
      Reduction in the incidence of type 2 diabetes with lifestyle intervention or metformin.
      The findings have broad implications because “prediabetes” affects approximately 88 million adults in the United States.
      Centers for Disease Control and Prevention (CDC)
      Prediabetes - your chance to prevent type 2 diabetes.
      Strenuous calls to address the epidemic of diabetes with prevention
      • Chen L.
      • Magliano D.J.
      • Zimmet P.Z.
      The worldwide epidemiology of type 2 diabetes mellitus--present and future perspectives.
      ,
      • Herman W.H.
      • Zimmet P.
      Type 2 diabetes: an epidemic requiring global attention and urgent action.
      have been counterbalanced by concerns about the overmedicalization of prediabetes.
      • Yudkin J.S.
      • Montori V.M.
      The epidemic of pre-diabetes: the medicine and the politics.
      Almost 2 decades after publication of the DPP Study, it remains unclear how best to implement these interventions in such an overwhelmingly large, and mostly undiagnosed, population. A 2015 study examining a national sample of more than 17,000 working-age adults with prediabetes found that only 3.7% were receiving metformin.
      • Moin T.
      • Li J.
      • Duru O.K.
      • et al.
      Metformin prescription for insured adults with prediabetes from 2010 to 2012: a retrospective cohort study.
      Similarly, widespread use of the intensive lifestyle intervention remains largely unrealized despite evidence that rigorous diet and physical activity promotion reduces diabetes risk in the community setting.
      • Balk E.M.
      • Earley A.
      • Raman D.
      • Avendano E.A.
      • Pittas A.G.
      • Remington P.L.
      Combined diet and physical activity promotion programs to prevent type 2 diabetes among persons at increased risk: a systematic review for the Community Preventive Services Task Force.
      However, prediabetes is itself a heterogeneous condition. We previously showed that even among patients enrolled in the DPP Study itself, the risk for developing diabetes within 3 years varies widely and is highly skewed.
      • Sussman J.B.
      • Kent D.M.
      • Nelson J.P.
      • Hayward R.A.
      Improving diabetes prevention with benefit based tailored treatment: risk based reanalysis of Diabetes Prevention Program.
      Some trial participants were estimated to have a 1% to 2% risk; others, 90%. Unsurprisingly, the degree of benefit from metformin therapy or from the lifestyle intervention was also distributed unevenly.
      This prior proof-of-concept work had several limitations. Notably, the risk distribution within the DPP trial participants may differ from that of patients seen in routine practice, particularly since the American Diabetes Association (ADA) has subsequently broadened its definition of prediabetes to include a still more heterogeneous population.
      American Diabetes Association
      Diagnosis and classification of diabetes mellitus.
      Further, the application of prediction methods to data routinely collected in the electronic health record (EHR) provides a promising means to overcome some of the major barriers to the use of risk models.
      • Watson J.
      • Hutyra C.A.
      • Clancy S.M.
      • et al.
      Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers?.
      ,
      • Wallace E.
      • Johansen M.E.
      Clinical prediction rules: challenges, barriers, and promise.
      For example, in addition to requiring manual ascertainment of variables, the previously reported DPP-based model required waist circumference and waist to hip ratio measurements that are not difficult to ascertain in routine practice. We describe the development of a clinical prediction model, the Tufts–Predictive Analytics and Comparative Effectiveness DPP risk model, using a hybrid approach that makes use of routinely collected EHR data to predict the risk for diabetes onset and clinical trial data to estimate unbiased risk-based effects of preventive interventions.

      Patients and Methods

       Overview

      We sought to develop and validate a diabetes risk prediction model using data elements readily available in the EHR for dissemination across health care systems as an EHR-embedded tool, to facilitate ease of use. The tool provides clinicians and their patients with an individualized risk for developing diabetes and the estimated benefit of applying a DPP treatment strategy, either an intensive lifestyle program or pharmacotherapy with metformin (the combination of both was not tested in the DPP Study).

       Data Sources and Participants

      The model was developed and validated using EHR data from the OptumLabs Data Warehouse (OLDW). The OptumLabs EHR database is a geographically diverse sample of the US population with longitudinal clinical data on more than 33 million lives with at least 1 clinic visit during the study period. Using a retrospective observational cohort design, we geographically stratified the database by US Census Region into a development cohort of 1,076,983 patients (Northeast, South, and West) and a separate validation cohort of 1,075,833 patients (Midwest).
      Eligibility criteria included age between 25 and 75 years on an “index” office or clinic encounter (index visit defined by Current Procedural Terminology/Healthcare Common Procedure Coding System codes; Supplemental Table 1, available online at http://www.mayoclinicproceedings.org) between January 1, 2012, and December 31, 2016, at which time they met laboratory-based criteria for the diagnosis of prediabetes. (The age enrollment was selected because it approximated the age distribution of the DPP trial, which enrolled patients ≥ 25 years.) Prediabetes was defined by current ADA criteria, that is, having no diagnosis of type 1 or type 2 diabetes on the problem list and one of the following within 12 months before the visit: hemoglobin A1c (HbA1c) level between 5.7% and 6.4% inclusive and/or fasting glucose (FG) level between 100 and 125 mg/dL (to convert to mmol/L, multiply by 0.0555) inclusive. Because labeling of fasting status may be incomplete, a glucose level drawn at the same time as a lipid panel or triglycerides was considered as fasting. We did not use the 2-hour post–glucose load criterion because it is rarely used in clinical practice for prediabetes. Patients were excluded if they had random (nonfasting) glucose levels of 200 mg/dL or greater on 2 occasions within a 3-month period before the index visit. Women with documented pregnancy within 24 months of the index visit were also excluded. To ascertain the development of diabetes, patients also had to have some clinical activity 3 years after the index visit. Eligibility criteria are detailed in Supplemental Table 1.
      The DPP data set was used to estimate treatment effect for metformin or the intensive lifestyle modification program. The design, rationale, outcomes, and loss to follow-up of the DPP have been described in detail elsewhere.
      • Knowler W.C.
      • Barrett-Connor E.
      • Fowler S.E.
      • et al.
      Diabetes Prevention Program Research Group
      Reduction in the incidence of type 2 diabetes with lifestyle intervention or metformin.
      ,
      Diabetes Prevention Program Research Group
      Design and methods for a clinical trial in the prevention of type 2 diabetes.
      Briefly, inclusion criteria included a body mass index (BMI; calculated as the weight in kilograms divided by the height in meters squared) of 24 or higher (≥22 kg/m2 in Asians) and a plasma FG concentration of 95 to 125 mg/dL inclusive (impaired FG) and a concentration of 140 to 199 mg/dL inclusive 2 hours after a 75-g oral glucose load (impaired glucose tolerance). We note that these criteria differ from the ADA’s current diagnostic criteria for prediabetes that we used for the OLDW model; the ADA definition imposes no BMI requirement.
      American Diabetes Association
      Standards of medical care in diabetes--2014.
      The DPP participants were randomly assigned to: (1) standard lifestyle recommendations plus 850 mg of metformin twice daily, (2) an intensive program of lifestyle modification that included 16 lessons with a case manager and set goals of at least a 7% weight loss and at least 150 minutes of physical activity per week, or (3) standard lifestyle recommendations plus placebo twice daily. After a median follow-up period of 2.8 (range, 1.8-4.6) years, progression to diabetes was reduced by 58% (95% CI, 47% to 66%) in the lifestyle modification arm and 31% (17% to 43%) in the metformin arm, both compared with the placebo arm.
      • Knowler W.C.
      • Barrett-Connor E.
      • Fowler S.E.
      • et al.
      Diabetes Prevention Program Research Group
      Reduction in the incidence of type 2 diabetes with lifestyle intervention or metformin.
      The National Institute of Diabetes and Digestive and Kidney Diseases Data Repository, from which we obtained data, includes 3081 of the 3234 DPP participants (95% of full population) because some local institutional review boards (IRBs) declined to participate in data distribution.

       Outcome

      For the OLDW cohort, the time-to-event outcome was defined as the time to the first patient encounter after the index visit with documented evidence of type 2 diabetes by any of the following criteria,
      • McCoy R.G.
      • Nori V.S.
      • Smith S.A.
      • Hane C.A.
      Development and Validation of HealthImpact: An Incident Diabetes Prediction Model Based on Administrative Data.
      diagnosis codes International Classification of Diseases, Ninth Revision 250.x0 or 250.x2 or International Classification of Diseases, Tenth Revision E11.xx, pharmacotherapy or procedure for type 2 diabetes (as detailed in Supplemental Table 1), HbA1c level greater than 6.4%, FG (or presumed fasting, as noted) level greater than 125 mg/dL, or 2-hour oral glucose tolerance test postload glucose level greater than 199 mg/dL. Laboratory-based criteria required confirmation by an additional laboratory in the diabetes range or by another method (ie, diagnosis or medication). Follow-up time for patients who did not meet the outcome definition was censored at the first occurrence of the last observed encounter or end of study period.

       Candidate Predictors

      A priori risk model predictors were identified by a systematic review conducted by Collins et al.
      • Collins G.S.
      • Mallett S.
      • Omar O.
      • Yu L.M.
      Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting.
      We selected the following 11 independent variables that were included in at least 3 prior diabetes risk models and were judged to be easily and reliably obtainable in EHR data: age, sex, race, smoking status, BMI, presence or absence of a diagnosis of hypertension, systolic blood pressure, high-density lipoprotein cholesterol level, triglyceride level, FG level, and HbA1c level. Four variables included in 3 prior models were not considered based on the difficulty of ascertaining them in EHR data: physical activity, waist circumference, waist to hip ratio, and family history of diabetes.

       Missing Data

      Missing data is a common limitation when working with EHR data.
      • Wells B.J.
      • Chagin K.M.
      • Nowacki A.S.
      • Kattan M.W.
      Strategies for handling missing data in electronic health record derived data.
      Although multiple imputation may improve estimates of parameter effects under a missing-at-random assumption, it does not provide a practical means to cope with missingness in actual patients for whom a prediction needs to be made. Thus, we used missing indicator variables to capture the predictive effects of missingness under the assumption that future and prior missingness are similarly informative. For each predictor, an additional dichotomous variable indicated the presence of missing values.
      • Sisk R.
      • Lin L.
      • Sperrin M.
      • et al.
      Informative presence and observation in routine health data: a review of methodology for clinical risk prediction.
      ,
      • Groenwold R.H.H.
      Informative missingness in electronic health record systems: the curse of knowing.
      For continuous variables (eg, BMI and HbA1c level), the missing value of the original variable was replaced by a fixed constant (the median) before model estimation, and the missing indicator variable appropriately adjusted for the “missing variable effect.” For categorical variables (eg, race and smoking status), an additional level was added to define the missing category.

       Model Development

      We used multivariable Cox proportional hazards regression to estimate the predicted probability of developing type 2 diabetes. We included 2 a priori interactions, race × BMI and race × HbA1c level, based on clinical judgment and the literature.
      • Beck R.W.
      • Riddlesworth T.D.
      • Ruedy K.
      • et al.
      DIAMOND Study Group
      Continuous glucose monitoring versus usual care in patients with type 2 diabetes receiving multiple daily insulin injections: a randomized trial.
      ,
      • Zhu Y.
      • Sidell M.A.
      • Arterburn D.
      • et al.
      Racial/ethnic disparities in the prevalence of diabetes and prediabetes by BMI: Patient Outcomes Research To Advance Learning (PORTAL) multisite cohort of adults in the U.S.
      Model performance was assessed for discrimination and calibration. A bootstrap resampling procedure with 500 samples was used to internally validate the model, estimate optimism-corrected discrimination, and assess calibration.

       Model Validation

      Using the equation derived in the development cohort, we calculated the predicted probability of developing type 2 diabetes for patients in the validation cohort. Model performance on external validation was assessed for discrimination using Harrell’s measure of concordance for censored response variable and calibration.
      • Harrell F.E.
      Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis.

       Estimating Risk-Specific Treatment Effects

      To estimate the risk-based treatment effect for metformin pharmacotherapy or the DPP lifestyle modification, we performed a risk-based heterogeneity of treatment effect analysis on the DPP.
      • Kent D.M.
      • Steyerberg E.W.
      • van Klaveren D.
      Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects.
      The applicability of the OLDW model to the DPP data was anticipated to be limited by differences between predictor variable definitions and measurement within a trial context vs EHR data, differences in the pattern of missingness between these contexts (ie, there was essentially no data missingness in the DPP), differences in patient enrollment in the 2 settings, and differences in outcome definition and ascertainment.
      • Luijken K.
      • Groenwold R.H.H.
      • Van Calster B.
      • Steyerberg E.W.
      • van Smeden M.
      Impact of predictor measurement heterogeneity across settings on the performance of prediction models: a measurement error perspective.
      Thus, we refit the OLDW model to the DPP, using the same variables and interaction terms. Consistent with methodological recommendations,
      • Kent D.M.
      • Paulus J.K.
      • van Klaveren D.
      • et al.
      The Predictive Approaches to Treatment Effect Heterogeneity (PATH) statement.
      ,
      • Kent D.M.
      • van Klaveren D.
      • Paulus J.K.
      • et al.
      The Predictive Approaches to Treatment Effect Heterogeneity (PATH) statement: explanation and elaboration.
      all 3 DPP arms were used because research has shown that overfitting to a control arm can induce spurious heterogeneity of treatment effects.
      • Abadie A.
      • Chingos M.
      • West M.
      Endogenous stratification in randomized experiments.
      • Burke J.F.
      • Hayward R.A.
      • Nelson J.P.
      • Kent D.M.
      Using internally developed risk models to assess heterogeneity in treatment effects in clinical trials.
      • van Klaveren D.
      • Balan T.A.
      • Steyerberg E.W.
      • Kent D.M.
      Models with interactions overestimated heterogeneity of treatment effects and were prone to treatment mistargeting.
      The treatment effect was then estimated by incorporating this linear predictor into a Cox proportional hazards model with the following terms: treatment (metformin or DPP lifestyle modification), the linear predictor of risk from the refitted model, and (potentially) an interaction between these to account for important changes in relative risk reduction across different levels of baseline risk. Based on a previous analysis,
      • Sussman J.B.
      • Kent D.M.
      • Nelson J.P.
      • Hayward R.A.
      Improving diabetes prevention with benefit based tailored treatment: risk based reanalysis of Diabetes Prevention Program.
      we anticipated a risk-by-treatment interaction with metformin pharmacotherapy and a consistent relative effect with the DPP lifestyle modification, but we examined interactions for both treatment arms. We also performed a sensitivity analysis, examining the risk-by-treatment interactions, stratifying the DPP by the OLDW model without any refitting, and examining the distribution of predicted effects using this model.

       Incorporation of Decision Support in EHR

      To facilitate use in clinical decision making, based on patient and provider focus groups and interviews, we implemented the model in 2 different ways: (1) a hard coded calculation in an Allscripts EHR, and (2) a cloud-hosted SMART on FHIR
      • Mandel J.C.
      • Kreda D.A.
      • Mandl K.D.
      • Kohane I.S.
      • Ramoni R.B.
      SMART on FHIR: a standards-based, interoperable apps platform for electronic health records.
      app that can be incorporated into any EHR, leveraging interoperability standards recently promulgated by the US office of the National Coordinator of Health Information.

       IRB Approval

      This study was reviewed and approved by the Tufts Health Sciences IRB before accessing the deidentified data from the DPP and OLDW data sets.

      Results

      Figure 1 shows the development of the derivation and validation OLDW data sets. Approximately 1.1 million people with prediabetes from the Northeast, South, and West were included in the derivation cohort, and a similar number from the Midwest were included in the validation cohort. Characteristics of these cohorts are shown in Table 1.
      Figure thumbnail gr1
      Figure 1Consolidated Standards of Reporting Trials diagram for OptumLabs Data Warehouse derivation and validation cohort. The figure depicts the number of eligible patients as each inclusion criterion is applied. Starting with an initial population size of more than 33 million patients, our final study cohort consists of 2,152,816 eligible patients.
      Table 1Cohort Characteristics
      DPP = Diabetes Prevention Program; OLDW = OptumLabs Data Warehouse; SD = standard deviation.
      ,
      SI conversion factors: To convert HDL cholesterol values to mmol/L, multiply by 0.0259; to convert glucose values to mmol/L, multiply by 0.0555; to convert triglyceride values to mmol/L, multiply by 0.0113.
      OverallOLDWDPP
      Missing, %n=2,152,816DevelopmentValidation
      n=1,076,983n=1,075,833n=3081
      Age (y), mean ± SD0.054.9±11.755.1±11.955±11.550.6±9.0
      Female sex, %0.150.351.349.166.6
      Race, %8.2
       White86.584.388.957.4
       Black10.210.89.120.9
       Other non-White race (Optum = Asian)3.44.91.95.2
      Smoking status, %15.6
       Current smoker23.320.026.49.0
       Never smoked48.053.242.935.2
       Former smoker28.826.830.755.8
      Height (cm), mean ± SD15.9170.1±10.1169.5±10.1170.7±10166.8±9.2
      Body mass index (kg/m2), mean ± SD12.231.1±730.8±6.731.8±6.933.5±5.8
      Diagnosis of hypertension, %044.544.445.027.1
      Systolic blood pressure (mm Hg), mean ± SD9.0127.4±14.9127.6±15.2127.3±14.7124.2±14.7
      HDL cholesterol (mg/dL), mean ± SD12.350.9±14.751.3±14.950.6±14.545.6±11.8
      Triglycerides (mg/dL), mean ± SD12.6138.3±72.8136.9±72.8139.7±72.7162.9±93.5
      Hemoglobin A1c (%), mean ± SD54.75.8±0.35.8±0.35.8±0.35.9±0.5
      Fasting plasma glucose (mg/dL), mean ± SD3.8103.7±10.8103±11.1104.5±10.4107.2±7.7
       Fasting plasma glucose, (fasting) (mg/dL), mean ± SD86.3103.3±9.2101.3±10.5105.3±7.3
      Fasting plasma glucose (random) (mg/dL), mean ± SD13.0103.7±11.4103.1±11.4104.4±11.2
      a DPP = Diabetes Prevention Program; OLDW = OptumLabs Data Warehouse; SD = standard deviation.
      b SI conversion factors: To convert HDL cholesterol values to mmol/L, multiply by 0.0259; to convert glucose values to mmol/L, multiply by 0.0555; to convert triglyceride values to mmol/L, multiply by 0.0113.

       Model Development and Validation: Risk Stratification

      The coefficients for each of the variable and interaction terms included in the model are shown in Table 2. The optimism-corrected C statistic on the derivation sample was 0.73. When the model was tested on the validation cohort, the C statistic was slightly higher at 0.76. Calibration on the validation cohort was very good (Figure 2). Harrell’s E statistic was 1.63% and the calibration intercept and slope were −0.27 and 1.12, respectively. Among the 268,959 patients in the lowest-risk quartile, the predicted diabetes rate was 3.1% (95% CI, 3.0% to 3.2%), while the observed rate was 1.8% (95% CI, 1.7% to 1.9%); among the 268,958 patients in the highest-risk quartile, the predicted diabetes rate was 19.2% (95% CI, 18.6% to 19.9%), while the observed rate was 19.6% (95% CI, 19.4% to 19.8%).
      Table 2Final Model for Incident Diabetes
      AA = Black/African American; BMI = body mass index; HbA1c = hemoglobin A1c; HDL = high-density lipoprotein.
      ,
      SI conversion factors: To convert HDL cholesterol values to mmol/L, multiply by 0.0259; to convert glucose values to mmol/L, multiply by 0.0555; to convert triglyceride values to mmol/L, multiply by 0.0113.
      ,
      Baseline hazard at S years (S0): 1 year = 0.02470, 2 years = 0.04757, 3 years = 0.07044.
      Hazard RatioLowerUpperHazard RatioLowerUpper
      Age, per 10 y1.081.081.08Adjustments for missing data
      Female sex1.211.191.23 Race (missing) vs White0.160.070.38
      Black vs White2.731.325.64 Smoking (missing) vs never1.081.061.11
      Asian vs White0.010.000.02 HbA1c (missing)0.750.740.77
      Current smoker vs never1.221.191.24 Fasting plasma glucose (missing)1.030.991.07
      Former smoker vs never1.111.091.13 Triglycerides (missing)1.081.031.12
      Hypertension1.231.211.25 BMI (missing)1.221.181.26
      HbA1c, per 0.1%1.241.181.29 Systolic blood pressure (missing)1.221.171.26
      Fasting plasma glucose, per 10 mg/dL1.291.291.29 HDL cholesterol (missing)1.231.171.28
      Triglycerides, per 10 mg/dL1.011.011.02 AA × BMI (missing)0.970.911.03
      BMI, per 5 units1.241.241.24 AA × HbA1c (missing)1.561.481.64
      Systolic blood pressure, per 20 mm Hg1.051.051.05 Asian × BMI (missing)0.770.700.84
      HDL cholesterol, per 10 mg/dL0.850.850.85 Asian × HbA1c (missing)2.031.852.23
      Black × BMI0.980.980.99 Race (missing) × BMI0.990.991.00
      Black × HbA1c0.950.841.07 Race (missing) × BMI (missing)0.820.770.87
      Asian × BMI1.000.991.01 Race (missing) × HbA1c1.401.221.61
      Asian × HbA1c2.321.912.83 Race (missing) × HbA1c (missing)1.461.371.55
      a AA = Black/African American; BMI = body mass index; HbA1c = hemoglobin A1c; HDL = high-density lipoprotein.
      b SI conversion factors: To convert HDL cholesterol values to mmol/L, multiply by 0.0259; to convert glucose values to mmol/L, multiply by 0.0555; to convert triglyceride values to mmol/L, multiply by 0.0113.
      c Baseline hazard at S years (S0): 1 year = 0.02470, 2 years = 0.04757, 3 years = 0.07044.
      Figure thumbnail gr2
      Figure 2Calibration curves. The figure on the left depicts the observed vs predicted 3-year rate of developing diabetes in the 1,076,983 million patients in the derivation cohort (Northeast, South, and West regions) divided into equal-sized tenths. The figure on the right depicts the observed vs predicted 3-year rate of developing diabetes in the 1,075,833 million patients in the validation cohort (Midwest). OLDW = OptumLabs Data Warehouse.

       Calculation of Relative Treatment Effects in the DPP Study

      Prior work demonstrated a consistent relative treatment effect across risk groups with the DPP lifestyle modification and an increasing relative effect with progressively higher risk for metformin pharmacotherapy.
      • Sussman J.B.
      • Kent D.M.
      • Nelson J.P.
      • Hayward R.A.
      Improving diabetes prevention with benefit based tailored treatment: risk based reanalysis of Diabetes Prevention Program.
      Using the OLDW model refit to the DPP data (Supplemental Table 2, available online at http://www.mayoclinicproceedings.org; C statistic, 0.719), we confirmed the absence of a treatment-by-risk interaction for lifestyle modification (P for interaction = .68); thus, we applied a constant relative risk reduction in the prediction model (hazard ratio [HR], 0.43; 95% CI, 0.35 to 0.53) to estimate the diabetes outcome with lifestyle modification. We also confirmed the presence of a treatment-by-risk interaction with metformin pharmacotherapy (P for interaction = .003; using the continuous risk on the logit scale): low-risk patients had outcomes with metformin that were similar to usual care (in lowest-risk quarter, observed HR, 1.1; 95% CI, 0.61 to 2.0), and high-risk patients have outcomes with metformin that were similar to the DPP lifestyle modification (in highest-risk quarter, observed HR, 0.45; 95% CI, 0.35 to 0.59).
      Figure 3 shows observed and predicted benefits across quartiles for the DPP for both lifestyle and metformin therapy. A look-up table showing the relative risk reduction with metformin for each level of risk is shown in Supplemental Table 3 (available online at http://www.mayoclinicproceedings.org), truncated at a low value of 0% (no harm or benefit) and a high value of 60%.
      Figure thumbnail gr3
      Figure 3Observed and predicted treatment effects in the Diabetes Prevention Program (DPP) Study across risk groups. Green dot and bar (95% confidence interval) are observed treatment effect. Blue dot and bar are predicted treatment effect. depicts the observed treatment effects (green dots) in patients in the DPP Study when patients are stratified into quarters based on predicted risk for the DPP lifestyle modification intervention (left) and for metformin (right). Predicted effects across risk groups are shown in blue. The top set of graphs displays relative effects and shows a consistency of effects across risk groups for lifestyle modification but heterogeneous treatment effects for metformin (P=.003). The bottom graphs show effects on the absolute risk difference scale, which shows increasing benefits for higher-risk patients for both interventions.

       Distribution of Risks and Benefits in OLDW

      The overall average 3-year predicted risk for developing diabetes for patients in the validation OLDW cohort was 9.0%, 3.9%, and 6.0% with usual care, the DPP lifestyle diabetes, and metformin therapy, respectively. For lifestyle modification, 53% of the total preventable cases of diabetes could be prevented by treating the 25% of patients at highest risk; 76%, by treating the 50% at highest risk; and 91%, by treating the 75% at highest risk. For metformin therapy, 73% of the total preventable cases could be prevented by treating the 25% of patients at highest risk; 93%, by treating the 50% at highest risk; and 100%, by treating the 75% at highest risk.

       Sensitivity Analyses

      Direct application of the OLDW model (not refit) on the DPP showed a moderately diminished discrimination (C statistic = 0.68). There was no risk-by treatment interaction with lifestyle (P=.69). The risk-by-treatment interaction with metformin therapy was qualitatively similar to that with the refit model (P=.08), and the distribution of predicted benefits with this model was also similar. For lifestyle modification, 53% of the total cases of preventable diabetes could be prevented by treating the 25% of patients at highest risk; 76%, by treating the 50% at highest risk. For metformin therapy, 65% of the total cases of preventable diabetes could be prevented by treating the 25% of patients at highest risk; 86%, by treating the 50% at highest risk.

       Implementation of the Final Model

      Figure 4 shows the user interface of the SMART app in an EHR. Predictions are generated automatically based on the data available and retrieved from the patient’s record, using appropriate indicators in the model for missingness when necessary.
      Figure thumbnail gr4
      Figure 4User interface for decision support tool. This figure depicts the interface of a clinical decision support tool currently implementing the Tufts–Predictive Analytics and Comparative Effectiveness Diabetes Prevention Program (DPP) risk model in clinical care. EHR = electronic health record; HDL = high-density lipoprotein; HTN = hypertension. SI conversion factors: To convert HDL cholesterol values to mmol/L, multiply by 0.0259; to convert glucose values to mmol/L, multiply by 0.0555; to convert triglyceride values to mmol/L, multiply by 0.0113.

      Discussion

      We present the Tufts–Predictive Analytics and Comparative Effectiveness DPP risk model, an EHR-compatible model that predicts diabetes onset based on 11 variables routinely collected in clinical practice. A major strength of the risk model is that it was derived on the OLDW, which reflects people with prediabetes defined by the most commonly used ADA criteria, from heterogeneous EHRs and more than 30 US health care systems. The risk model derived in 3 US Census regions performed very well in a geographically distinct cohort. Compatible risk-specific estimates of treatment effect were then obtained directly from the DPP. By prioritizing care based on the risk for diabetes, this “hybrid” model might help optimize the efficiency of diabetes prevention: treating just the highest-risk half of people with prediabetes would capture 77% of the benefit of population-wide lifestyle modification or 93% of the benefit of population-wide metformin pharmacotherapy. This is important because lifestyle programs are resource intensive and require a high level of commitment from the patient. Pharmacotherapy is not without adverse effects and overtreatment should be avoided, especially in low-risk patients who do not appear to benefit.
      The issue of how to address prediabetes has grown in importance as broader diabetes screening has been recommended and promoted.
      American Diabetes Association
      Standards of medical care in diabetes--2014.
      ,
      • Siu A.L.
      U S Preventive Services Task Force
      Screening for abnormal blood glucose and type 2 diabetes mellitus: U.S. Preventive Services Task Force Recommendation Statement.
      For every patient with diabetes identified, screening identifies 6 patients with prediabetes; health systems are thus confronted with a growing number of patients who have prediabetes, without the capacity to treat everybody, reserving limited resources to improving cardiometabolic control for patients with diabetes.
      Although the ADA has lowered the HbA1c and FG thresholds to define prediabetes,
      American Diabetes Association
      Diagnosis and classification of diabetes mellitus.
      ,
      • Genuth S.
      • Alberti K.G.
      • Bennett P.
      • et al.
      Expert Committee on the Diagnosis and Classification of Diabetes Mellitus. Follow-up report on the diagnosis of diabetes mellitus.
      some have argued that the value of medicalizing prediabetes and defining an ever-growing proportion of the population as diseased is of dubious value.
      • Yudkin J.S.
      • Montori V.M.
      The epidemic of pre-diabetes: the medicine and the politics.
      Most patients who are classified as prediabetic do not develop diabetes even in a decade, and risks for developing end-organ damage are low for those developing diabetes later in life.
      • Vijan S.
      • Sussman J.B.
      • Yudkin J.S.
      • Hayward R.A.
      Effect of patients' risks and preferences on health gains with plasma glucose level lowering in type 2 diabetes mellitus.
      Risk stratification offers an approach that promises more focused resources specifically on those who are likely to benefit. Although our prior research results provided proof of concept that risk stratification could support providers and health systems prioritize these patients,
      • Sussman J.B.
      • Kent D.M.
      • Nelson J.P.
      • Hayward R.A.
      Improving diabetes prevention with benefit based tailored treatment: risk based reanalysis of Diabetes Prevention Program.
      the present EHR-compatible model is designed to be used at point of care, and it has been incorporated into the EHRs at several locations in the United States.
      A long-standing concern regarding limitations of randomized clinical trial results is that they might not be applicable to “real-world” populations when there is nonrandom selection into the trial and treatment effects are heterogeneous.
      • Longford N.T.
      Selection bias and treatment heterogeneity in clinical trials.
      Here, for example, we found that the real-world at-risk population was at substantially lower overall risk than patients included in the DPP and that treatment effects were risk dependent. The lower overall risk in the OLDW cohort is the result of multiple factors, including: (1) different inclusion criteria for the DPP (including a high BMI and elevated 2-hour glucose level after a 75-g glucose load), (2) differences in the distribution of risk variables (see Table 1), and (3) different outcome ascertainment, which is substantially more rigorous in the trial setting. Cross-design synthesis has been proposed as a means of addressing the potential problems of external validity of trial evidence by combining the strengths of both designs; observational designs to capture the full range of patients and randomized trials for unbiased treatment effects.
      • Droitcour J.
      • Silberman G.
      • Chelimsky E.
      Cross-design synthesis: a new form of meta-analysis for combining results from randomized clinical trials and medical-practice databases.
      ,
      • Kaizar E.E.
      Estimating treatment effect via simple cross design synthesis.
      Beyond differences in risk, a related concern is whether the relative effects seen in the trial would apply in routine clinical care, for which the patient/provider commitment may be less than ideal. We believe it is appropriate to provide these estimates in shared decision making because they estimate effects that patients should anticipate if they have trial-like adherence to the interventions.
      Although several different methods for cross-design synthesis have been proposed,
      • Cole S.R.
      • Stuart E.A.
      Generalizing evidence from randomized clinical trials to target populations.
      ,
      • Varadhan R.
      • Henderson N.C.
      • Weiss C.O.
      Cross-design synthesis for extending the applicability of trial evidence when treatment effect is heterogeneous: part I. Methodology.
      all approaches depend on the ability to adjust results based on patient characteristics across designs. A seldom discussed barrier is that variable definitions and ascertainment can differ considerably between clinical trial data and routinely collected observational data. Our approach was designed to address these barriers in a pragmatic way, by estimating risk-specific treatment effects in the clinical trial using the same set of variables as used in the observational risk model. This approach was driven in part by our novel aim, to predict effects in patients in clinical care based on individual patient characteristics, rather than estimating average treatment effects in broad target populations.
      A related issue that has received limited attention is how to deploy clinical prediction models in an EHR. There is a proliferation of clinical prediction models; use of routinely collected EHR data to automatically generate individual patient predictions is an appealing approach to disseminate these into the clinic. However, most published clinical prediction models are developed on research cohorts or clinical trials. Predictor variables collected in a trial are not consistently and rigorously captured in the EHR. Recent work has highlighted that heterogeneity in predictor measurement across different settings can substantially degrade model performance.
      • Luijken K.
      • Groenwold R.H.H.
      • Van Calster B.
      • Steyerberg E.W.
      • van Smeden M.
      Impact of predictor measurement heterogeneity across settings on the performance of prediction models: a measurement error perspective.
      ,
      • Luijken K.
      • Wynants L.
      • van Smeden M.
      • Van Calster B.
      • Steyerberg E.W.
      • Groenwold R.H.H.
      • Collaborators
      Changing predictor measurement procedures affected the performance of prediction models in clinical examples.
      Finally, use of trial or registry data cannot yield a model robust to missing values in the EHR database used for clinical prediction because the pattern of missingness present across research and EHR environments is expected to differ. The usual approaches addressing potential bias arising from missingness (eg, multiple imputation) are not designed to cope with missingness in variables used to generate predictions. These issues guided our decision to derive separate models in the EHR and trial setting, using a common set of variables that were well ascertained in both settings.
      There are some limitations. The methods we used for “cross-walking” between the 2 very different types of data (trial and EHR real-world data) potentially introduce estimation error. Ideally, individualized treatment effects would be estimated on databases that combine the advantages of these different data sources: unbiased effect estimates through randomization, meticulous outcome ascertainment, consistency of predictors across derivation and implementation populations, and large heterogeneous populations. Improving the quality of data collection in routine care and integrating randomized trials into routine care
      • Vickers A.J.
      • Scardino P.T.
      The clinically-integrated randomized trial: proposed novel method for conducting large trials at low cost.
      • Simon K.C.
      • Tideman S.
      • Hillman L.
      • et al.
      Design and implementation of pragmatic clinical trials using the electronic medical record and an adaptive design.
      • van Staa T.P.
      • Dyson L.
      • McCann G.
      • et al.
      The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials.
      may narrow the gap between trial and real-world data. That oral glucose tolerance testing was used both for entry criteria and end point ascertainment presumably in the DPP trial contributed to the higher risk in the DPP trial cohort compared with the OLDW cohort. Lorenzo et al
      • Lorenzo C.
      • Wagenknecht L.E.
      • Hanley A.J.
      • Rewers M.J.
      • Karter A.J.
      • Haffner S.M.
      A1C between 5.7 and 6.4% as a marker for identifying pre-diabetes, insulin sensitivity and secretion, and cardiovascular risk factors: the Insulin Resistance Atherosclerosis Study (IRAS).
      demonstrated that the sensitivity of HbA1c and/or FG levels in the diagnosis of prediabetes and diabetes is relatively low (76% and 52%, respectively). Because the results of oral glucose tolerance testing are not generally available for most patients in routine care, we were unable to adjust for these differences. Conversely, incorporating the use of pharmacotherapy into our identification of the outcome may have caused some misclassification of patients without diabetes. However, we anticipate that this rate is very low. There were also other variables known to be predictors of diabetes onset (eg, waist to hip ratio) that are not well collected in routine care and so were not considered for our model. Finally, although the OLDW is representative of the commercially insured population, some caution is recommended in extrapolating the results beyond this.
      Despite these limitations, we obtained qualitatively consistent risk-stratified results in the DPP regardless of which risk model was used: consistency of relative treatment effects of lifestyle modification across all levels of risk and heterogeneous relative treatment effects with metformin, with much stronger relative effects in higher-risk patients.

      Conclusion

      Although the number of people in the United States who have prediabetes and qualify for diabetes prevention programs could potentially overwhelm health care systems, these patients have substantial variation in their risk for developing diabetes and in their likelihood of benefiting from prevention therapies. Incorporation of a tool into the EHR to support automated risk stratification of patients in routine clinical care, by predicting individualized benefits, can support shared decision making and prioritize patients who are most likely to benefit, when capacity might be limited.

      Acknowledgments

      Research reported in this publication was funded through a Patient-Centered Outcomes Research Institute award (DI-1604-35234). The statements in this publication are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute, its Board of Governors, or Methodology Committee. A.G. Pittas is supported in part by generous donations to the Tupper Research Fund at Tufts Medical Center.

      Supplemental Online Material

      References

        • Knowler W.C.
        • Barrett-Connor E.
        • Fowler S.E.
        • et al.
        • Diabetes Prevention Program Research Group
        Reduction in the incidence of type 2 diabetes with lifestyle intervention or metformin.
        N Engl J Med. 2002; 346: 393-403
        • Centers for Disease Control and Prevention (CDC)
        Prediabetes - your chance to prevent type 2 diabetes.
        (Published 2020. Accessed October 16, 2020)
        • Chen L.
        • Magliano D.J.
        • Zimmet P.Z.
        The worldwide epidemiology of type 2 diabetes mellitus--present and future perspectives.
        Nat Rev Endocrinol. 2011; 8: 228-236
        • Herman W.H.
        • Zimmet P.
        Type 2 diabetes: an epidemic requiring global attention and urgent action.
        Diabetes Care. 2012; 35: 943-944
        • Yudkin J.S.
        • Montori V.M.
        The epidemic of pre-diabetes: the medicine and the politics.
        BMJ. 2014; 349: g4485
        • Moin T.
        • Li J.
        • Duru O.K.
        • et al.
        Metformin prescription for insured adults with prediabetes from 2010 to 2012: a retrospective cohort study.
        Ann Intern Med. 2015; 162: 542-548
        • Balk E.M.
        • Earley A.
        • Raman D.
        • Avendano E.A.
        • Pittas A.G.
        • Remington P.L.
        Combined diet and physical activity promotion programs to prevent type 2 diabetes among persons at increased risk: a systematic review for the Community Preventive Services Task Force.
        Ann Intern Med. 2015; 163: 437-451
        • Sussman J.B.
        • Kent D.M.
        • Nelson J.P.
        • Hayward R.A.
        Improving diabetes prevention with benefit based tailored treatment: risk based reanalysis of Diabetes Prevention Program.
        BMJ. 2015; 350: h454
        • American Diabetes Association
        Diagnosis and classification of diabetes mellitus.
        Diabetes Care. 2010; 33: S62-S69
        • Watson J.
        • Hutyra C.A.
        • Clancy S.M.
        • et al.
        Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers?.
        JAMIA Open. 2020; 3: 167-172
        • Wallace E.
        • Johansen M.E.
        Clinical prediction rules: challenges, barriers, and promise.
        Ann Fam Med. 2018; 16: 390-392
        • Diabetes Prevention Program Research Group
        Design and methods for a clinical trial in the prevention of type 2 diabetes.
        Diabetes Care. 1999; 22: 623-634
        • American Diabetes Association
        Standards of medical care in diabetes--2014.
        Diabetes Care. 2014; 37: S14-S80
        • McCoy R.G.
        • Nori V.S.
        • Smith S.A.
        • Hane C.A.
        Development and Validation of HealthImpact: An Incident Diabetes Prediction Model Based on Administrative Data.
        Health Serv Res. 2016; 51: 1896-1918
        • Collins G.S.
        • Mallett S.
        • Omar O.
        • Yu L.M.
        Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting.
        BMC Med. 2011; 9: 103
        • Wells B.J.
        • Chagin K.M.
        • Nowacki A.S.
        • Kattan M.W.
        Strategies for handling missing data in electronic health record derived data.
        EGEMS (Wash DC). 2013; 1: 1035
        • Sisk R.
        • Lin L.
        • Sperrin M.
        • et al.
        Informative presence and observation in routine health data: a review of methodology for clinical risk prediction.
        J Am Med Inform Assoc. 2021; 28: 155-166
        • Groenwold R.H.H.
        Informative missingness in electronic health record systems: the curse of knowing.
        Diagn Progn Res. 2020; 4: 8
        • Beck R.W.
        • Riddlesworth T.D.
        • Ruedy K.
        • et al.
        • DIAMOND Study Group
        Continuous glucose monitoring versus usual care in patients with type 2 diabetes receiving multiple daily insulin injections: a randomized trial.
        Ann Intern Med. 2017; 167: 365-374
        • Zhu Y.
        • Sidell M.A.
        • Arterburn D.
        • et al.
        Racial/ethnic disparities in the prevalence of diabetes and prediabetes by BMI: Patient Outcomes Research To Advance Learning (PORTAL) multisite cohort of adults in the U.S.
        Diabetes Care. 2019; 42: 2211-2219
        • Harrell F.E.
        Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis.
        Springer-Verlag, New York, NY2001
        • Kent D.M.
        • Steyerberg E.W.
        • van Klaveren D.
        Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects.
        BMJ. 2018; 363: k4245
        • Luijken K.
        • Groenwold R.H.H.
        • Van Calster B.
        • Steyerberg E.W.
        • van Smeden M.
        Impact of predictor measurement heterogeneity across settings on the performance of prediction models: a measurement error perspective.
        Stat Med. 2019; 38: 3444-3459
        • Kent D.M.
        • Paulus J.K.
        • van Klaveren D.
        • et al.
        The Predictive Approaches to Treatment Effect Heterogeneity (PATH) statement.
        Ann Intern Med. 2020; 172: 35-45
        • Kent D.M.
        • van Klaveren D.
        • Paulus J.K.
        • et al.
        The Predictive Approaches to Treatment Effect Heterogeneity (PATH) statement: explanation and elaboration.
        Ann Intern Med. 2020; 172: W1-W25
        • Abadie A.
        • Chingos M.
        • West M.
        Endogenous stratification in randomized experiments.
        Rev Econ Stat. 2018; 100: 567-580
        • Burke J.F.
        • Hayward R.A.
        • Nelson J.P.
        • Kent D.M.
        Using internally developed risk models to assess heterogeneity in treatment effects in clinical trials.
        Circ Cardiovasc Qual Outcomes. 2014; 7: 163-169
        • van Klaveren D.
        • Balan T.A.
        • Steyerberg E.W.
        • Kent D.M.
        Models with interactions overestimated heterogeneity of treatment effects and were prone to treatment mistargeting.
        J Clin Epidemiol. 2019; 114: 72-83
        • Mandel J.C.
        • Kreda D.A.
        • Mandl K.D.
        • Kohane I.S.
        • Ramoni R.B.
        SMART on FHIR: a standards-based, interoperable apps platform for electronic health records.
        J Am Med Inform Assoc. 2016; 23: 899-908
        • Siu A.L.
        • U S Preventive Services Task Force
        Screening for abnormal blood glucose and type 2 diabetes mellitus: U.S. Preventive Services Task Force Recommendation Statement.
        Ann Intern Med. 2015; 163: 861-868
        • Genuth S.
        • Alberti K.G.
        • Bennett P.
        • et al.
        Expert Committee on the Diagnosis and Classification of Diabetes Mellitus. Follow-up report on the diagnosis of diabetes mellitus.
        Diabetes Care. 2003; 26: 3160-3170
        • Vijan S.
        • Sussman J.B.
        • Yudkin J.S.
        • Hayward R.A.
        Effect of patients' risks and preferences on health gains with plasma glucose level lowering in type 2 diabetes mellitus.
        JAMA Intern Med. 2014; 174: 1227-1234
        • Longford N.T.
        Selection bias and treatment heterogeneity in clinical trials.
        Stat Med. 1999; 18: 1467-1474
        • Droitcour J.
        • Silberman G.
        • Chelimsky E.
        Cross-design synthesis: a new form of meta-analysis for combining results from randomized clinical trials and medical-practice databases.
        Int J Technol Assess Health Care. 1993; 9: 440-449
        • Kaizar E.E.
        Estimating treatment effect via simple cross design synthesis.
        Stat Med. 2011; 30: 2986-3009
        • Cole S.R.
        • Stuart E.A.
        Generalizing evidence from randomized clinical trials to target populations.
        Am J Epidemiol. 2010; 172: 107-115
        • Varadhan R.
        • Henderson N.C.
        • Weiss C.O.
        Cross-design synthesis for extending the applicability of trial evidence when treatment effect is heterogeneous: part I. Methodology.
        Commun Stat Case Stud Data Anal Appl. 2017; 2: 112-126
        • Luijken K.
        • Wynants L.
        • van Smeden M.
        • Van Calster B.
        • Steyerberg E.W.
        • Groenwold R.H.H.
        • Collaborators
        Changing predictor measurement procedures affected the performance of prediction models in clinical examples.
        J Clin Epidemiol. 2020; 119: 7-18
        • Vickers A.J.
        • Scardino P.T.
        The clinically-integrated randomized trial: proposed novel method for conducting large trials at low cost.
        Trials. 2009; 10: 14
        • Simon K.C.
        • Tideman S.
        • Hillman L.
        • et al.
        Design and implementation of pragmatic clinical trials using the electronic medical record and an adaptive design.
        JAMIA Open. 2018; 1: 99-106
        • van Staa T.P.
        • Dyson L.
        • McCann G.
        • et al.
        The opportunities and challenges of pragmatic point-of-care randomised trials using routinely collected electronic records: evaluations of two exemplar trials.
        Health Technol Assess. 2014; 18: 1-146
        • Lorenzo C.
        • Wagenknecht L.E.
        • Hanley A.J.
        • Rewers M.J.
        • Karter A.J.
        • Haffner S.M.
        A1C between 5.7 and 6.4% as a marker for identifying pre-diabetes, insulin sensitivity and secretion, and cardiovascular risk factors: the Insulin Resistance Atherosclerosis Study (IRAS).
        Diabetes Care. 2010; 33: 2104-2109