If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Correspondence: Address to Joseph D. Feuerstein, MD, Department of Medicine and Division of Gastroenterology, Beth Israel Deaconess Medical Center, Harvard Medical School, 330 Brookline Ave, E/DANA 501, Boston, MA 02215.
To determine the validity of guidelines published by interventional medical societies.
Methods
We reviewed the interventional medicine subspecialty society websites of the American Association for Bronchology and Interventional Pulmonology (AABIP), American Society of Diagnostic and Interventional Nephrology (ASDIN), American Society for Gastrointestinal Endoscopy (ASGE), and Society for Cardiovascular Angiography and Interventions (SCAI) as of November 15, 2012, for published interventional guidelines. The study was performed between November 15, 2012, and January 1, 2013. The AABIP did not publish guidelines, so American Thoracic Society and American College of Chest Physicians guidelines were reviewed. All the guidelines were reviewed for graded levels of evidence, methods used to grade the evidence, and disclosures of conflicts of interest (COIs).
Results
Of 153 interventional guidelines evaluated, 4 were duplicates. Forty-six percent of guidelines (69 of 149) graded the quality of evidence using 7 different methods. The ASGE graded 71% of guidelines (46 of 65) compared with 29% (23 of 78) by the SCAI and 0 by the ASDIN (n=4) and the pulmonary societies (n=2). Of the 3425 recommendations reviewed, 11% (n=364) were supported by level A, 42% (n=1432) by level B, and 48% (n=1629) by level C. The mean age of the guidelines was 5.2 years. Additionally, 62% of the guidelines (92 of 149) failed to comment on COIs; when disclosed, 91% of guidelines (52 of 57) reported COIs. In total, 1827 COIs were reported by 45% of the authors (317 of 697), averaging 5.8 COIs per author.
Conclusion
Most of the interventional guidelines failed to grade the evidence. When present, most guidelines used lower-quality evidence. Furthermore, most guidelines failed to disclose COIs. When commented on, numerous COIs were present. Future guidelines should clearly state the quality of evidence, use a standard grading system, be transparent regarding potential biases, and provide frequent updates.
The IOM defines a guideline as “systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances.”
(p.2) Guidelines are meant to create a succinct roadmap for the diagnosis and treatment of medical conditions by analyzing and summarizing the increasingly abundant medical literature. These guidelines have an effect on physicians and clinical practice and are also used by government organizations, insurance companies, and malpractices attorneys.
Guidelines are used as a means to establish a standard of care. This standard of care assumes that the guidelines are fundamentally sound and supply the framework for providing exceptional care. However, a guideline's validity is rooted in its development process.
Limitations occur when the evidence to support recommendations is lacking and recommendations are based solely on expert opinion. This becomes more worrisome when the authors have underlying conflicts of interest (COIs) that could bias recommendations.
Committee on Conflict of Interest in Medical Research Education and Practice, Institute of Medicine Conflict of Interest in Medical Research, Education, and Practice.
Prevalence of financial conflicts of interest among panel members producing clinical practice guidelines in Canada and United States: cross sectional study.
Previous studies have also noted issues regarding COIs in practice guidelines, finding that most authors of the cardiology guidelines had multiple COIs.
The primary goal of practice guidelines is to improve the safety and quality of care. One area that poses a relatively high risk of harm is procedures performed in internal medicine and its subspecialties. The medicine subspecialties of cardiology, gastroenterology, nephrology, and pulmonology all have interventional-specific societies. Many of these societies have developed intervention-specific guidelines to inform and standardize their society's procedure practices. Given the risks associated with interventional procedures, it is important that these guidelines are based on strong evidence.
We, therefore, performed a systematic review of medicine subspecialty interventional guidelines published on the professional websites of the American Society for Gastrointestinal Endoscopy (ASGE), the American Association of Bronchology and Interventional Pulmonology (AABIP), the American Society of Diagnostic and Interventional Nephrology (ASDIN), and the Society for Cardiovascular Angiography and Interventions (SCAI). The goal was to evaluate the quality of the evidence cited in formulating the recommendations, review the methods used to grade the evidence, assess potential COIs, and highlight opportunities for improvement.
Each guideline was reviewed to determine whether any grading system was used to assess the level of evidence for the recommendations. If a grading system was used, the level of evidence supporting the recommendations was evaluated. The layouts of the guidelines were assessed for consistency and easily identifiable recommendations. The age of the guidelines and any comments regarding planned updates to the current guidelines were also evaluated in the guideline document and on the societal website. The guidelines were examined individually, in aggregate by society, and between societies.
Levels of Evidence
The societies used multiple systems when grading the level of evidence. To standardize the reporting of the level of evidence, when possible, we merged the grading systems into the standard ABC grading system that has been used by multiple societies9,11: grade A, randomized controlled trials/meta-analyses; grade B, single randomized controlled/nonrandomized trials; and grade C, expert opinion/case studies/standard of care. Supplemental Table 1 (available online at http://www.mayoclinicproceedings.org) describes the methods used to standardize the grading systems.
Conflicts of Interest
All the guidelines were reviewed for any comments regarding COIs. We determined whether a disclosure was made noting COIs, a comment was made that no COIs were present, or there was no specific mention of COIs. If a COI was present, the guideline was further analyzed to calculate the number of authors with COIs, the number of COIs for the first author, and the number of COIs recorded per author. Conflicts of interest were subdivided into research awards/grants and others, including advisory board, speaker's bureau, consulting, industry-sponsored continuing medical education activities, and expert witnesses. Government and nonprofit-based research awards and volunteer work for nonprofit organizations were not considered COIs and were excluded. The COIs were evaluated by the individual society, between societies, and in aggregate.
Review of the Guidelines
The societal websites and the guidelines were reviewed by 3 of us (J.D.F., A.E.G., and C.M.H.) for the presence of any grading of the quality of evidence for the recommendations, the methods by which the evidence was graded, the clarity of the document layout, updates, and documentation of potential COIs. The merging of grading systems into the ABC format was performed by 2 of us (J.D.F. and A.S.C.).
Exclusions
Guidelines not posted on the societies' websites were not evaluated. When the evidence system used could not be consolidated into the ABC format, it was excluded from further subgroup analysis. Also, if 2 identical versions of a guideline with the same title were published, the earlier guideline was excluded from additional analysis. See Figure 1 for the inclusion and exclusion of guidelines.
Figure 1Process for the inclusion and exclusion of practice guidelines. AABIP = American Association for Bronchology and Interventional Pulmonology; ACCP/CHEST = American College of Chest Physicians; ASDIN = American Society of Diagnostic and Interventional Nephrology; ASGE = American Society for Gastrointestinal Endoscopy; ATS = American Thoracic Society; SCAI = Society for Cardiovascular Angiography and Interventions.
The χ2 test and the Fisher exact test were used for comparing proportions of graded evidence and COIs reported between societal guidelines for small and large samples, respectively. Mean years since publication was evaluated by 1-way analysis of variance. Pairwise comparisons were made when evaluating proportions across several categories, and Bonferroni corrections were used for multiple pairwise comparisons. A P≤.05 was considered significant. Analysis was performed using Stata (version 10.0, StataCorp LP).
Results
Guideline Grading of the Quality of Evidence
Of the interventional-specific societies reviewed, the ASDIN, the ASGE, and the SCAI published their own guidelines. A total of 153 interventional guidelines were reviewed as published on the societal websites of the American College of Chest Physicians, the ASGE, the ASDIN, the American Thoracic Society, and the SCAI. Two of the SCAI guidelines and 2 of the ASGE guidelines were duplicates and were excluded. Only 46% of the interventional society guidelines (69 of 149) graded the quality of evidence for their recommendations (Table 1 and Figure 1). The ASGE graded most of their clinical guidelines (71%; (46 of 65), whereas the SCAI graded 29% (23 of 78), and the ASDIN (n=4) and the pulmonary societies (n=2) failed to grade supporting levels of evidence (P<.001).
Table 1Practice Guidelines With Graded Evidence and Quality of Evidence for Recommendations
ACCP/CHEST = American College of Chest Physicians; ASDIN = American Society of Diagnostic and Interventional Nephrology; ASGE = American Society for Gastrointestinal Endoscopy; ATS = American Thoracic Society; NA = not applicable; SCAI = Society for Cardiovascular Angiography and Interventions.
Comparing ASGE with SCAI (ASDIN and ACCP/CHEST were taken out because they have 0 graded recommendations).
a ACCP/CHEST = American College of Chest Physicians; ASDIN = American Society of Diagnostic and Interventional Nephrology; ASGE = American Society for Gastrointestinal Endoscopy; ATS = American Thoracic Society; NA = not applicable; SCAI = Society for Cardiovascular Angiography and Interventions.
b Two duplicate guidelines from the ASGE and 2 duplicate guidelines from the SCAI were excluded from further analysis.
c One guideline was excluded from further analysis due to the inability to merge evidence.
d Seven guidelines were excluded from further analysis due to the inability to merge evidence.
e Comparing ASGE with SCAI (ASDIN and ACCP/CHEST were taken out because they have 0 graded recommendations).
Seven different methods were used to grade the quality of evidence in the ASGE and SCAI guidelines. The ASGE used 5 unique systems designed to grade the level of evidence and recommendations. In contrast, the SCAI used 2 systems. One of the 2 SCAI systems graded the level of evidence, and the second system, the appropriate use criteria (AUC), focused on panelists' opinions regarding the appropriateness of a test.
Levels of Evidence
Sixty-one guidelines (88%; 61 of 69) that graded the level of evidence were merged into the ABC system. Seven of the guidelines that could not be consolidated were published by the SCAI using the AUC system. One guideline from the ASGE used an evidence system that could not be merged appropriately (Supplemental Table 2; available online at http://www.mayoclinicproceedings.org). In total, 3425 recommendations graded the supporting level of evidence. Eleven percent of the recommendations (n=364) were supported by level A evidence, 42% (n=1432) by level B, and 48% (n=1629) by level C (Table 1 and Figure 2). When present, the levels of evidence did not differ by society (Table 1).
Figure 2Comparison of quality of evidence of recommendations in the (A) American Society for Gastrointestinal Endoscopy and (B) Society for Cardiovascular Angiography and Interventions guidelines with graded evidence. ∗Evidence was cited but was unable to be merged into the ABC level of evidence system. ∗∗The grading system used the appropriate use criteria and was unable to be Q7 merged into the ABC level of evidence system. ERCP ¼ endoscopic retrograde cholangiopancreatography; EUS ¼ endoscopic ultrasound; GERD ¼ gastroesophageal reflux disease; GI ¼ gastrointestinal; PCI ¼ percutaneous coronary intervention; STEMI ¼ ST-elevation myocardial infarction.
Figure 2Comparison of quality of evidence of recommendations in the (A) American Society for Gastrointestinal Endoscopy and (B) Society for Cardiovascular Angiography and Interventions guidelines with graded evidence. ∗Evidence was cited but was unable to be merged into the ABC level of evidence system. ∗∗The grading system used the appropriate use criteria and was unable to be Q7 merged into the ABC level of evidence system. ERCP ¼ endoscopic retrograde cholangiopancreatography; EUS ¼ endoscopic ultrasound; GERD ¼ gastroesophageal reflux disease; GI ¼ gastrointestinal; PCI ¼ percutaneous coronary intervention; STEMI ¼ ST-elevation myocardial infarction.
Ninety-six percent of the ASGE guidelines (44 of 46) that graded the level of evidence provided a summary paragraph at the end of the guideline listing all the recommendations. The SCAI guidelines that used the AUC system (n=7) had similar document layouts with clear tables including all the recommendations. The guidelines using other grading systems did not use standardized layouts to highlight the recommendations from the remainder of the document.
Methods Used to Update the Guidelines
The methods used to update guidelines varied by society. The ASGE and some of the SCAI updates removed any reference to older versions of the guidelines from their website. However, if searched for on PubMed (http://www.ncbi.nlm.nih.gov/pubmed), the outdated guideline did not have any notation indicating that the guideline is outdated and an update is present. The SCAI also performed multiple focused updates. This method left the original guideline and the focused update on the society's guidelines webpage. However, the guidelines that were being updated and the recommendations that were reclassified or removed were not deleted or annotated in the older document.
Age of Guidelines
The mean age of the guidelines, including any online updates on the societal websites, was 5.2 years. When analyzed by society, the mean age of the ASGE guidelines was 4.1 years; SCAI guidelines, 3.8 years; ASDIN guidelines, 3.3 years; and pulmonary guidelines, 9.5 years (P=.16). None of the guidelines published dates regarding the timing of planned revisions or updates.
Conflicts of Interest
Sixty-two percent of the guidelines (92 of 149) failed to comment on COIs. Of the 57 articles that made a disclosure regarding COIs, 52 (91%) reported authors with potential COIs. Of 1827 COIs reported, 861 were research based and 966 were other COIs. Forty-five percent of authors (317 of 697) involved in writing the guidelines reported an average of 5.8 conflicts. Forty-eight percent of first authors (25 of 52) reported an average of 3.5 COIs. When broken down by society, 47% of the SCAI guidelines (37 of 78) failed to comment on COIs compared with 75% of the ASGE guidelines (49 of 65) and 100% of the ASDIN (n=4) and pulmonary societies (n=2) guidelines (P<.001). Mean number of COIs per article and per first author did not differ between societies (Table 2).
Table 2Conflicts of Interest in Practice Guidelines
ACCP/CHEST = American College of Chest Physicians; ASDIN = American Society of Diagnostic and Interventional Nephrology; ASGE = American Society for Gastrointestinal Endoscopy; ATS = American Thoracic Society; COI = conflict of interest; NA = not applicable; SCAI = Society for Cardiovascular Angiography and Interventions.
Variable
Combined
ASGE
ASDIN
ACCP/CHEST and ATS
SCAI
P value
Articles with no comment on COIs (No./total No. [%])
a ACCP/CHEST = American College of Chest Physicians; ASDIN = American Society of Diagnostic and Interventional Nephrology; ASGE = American Society for Gastrointestinal Endoscopy; ATS = American Thoracic Society; COI = conflict of interest; NA = not applicable; SCAI = Society for Cardiovascular Angiography and Interventions.
which, in turn, depends on the strength of the guidelines. We noted that less than 50% of interventional society guidelines graded the level of evidence to support their recommendations. When graded, nearly 50% of the recommendations were based only on expert opinion. Furthermore, most of the guidelines failed to comment regarding any potential COIs. When COIs were mentioned, most guidelines contained numerous potential COIs.
Professional medical societies view practice guidelines as some of the most important services they provide.
Committee on Conflict of Interest in Medical Research Education and Practice, Institute of Medicine Conflict of Interest in Medical Research, Education, and Practice.
Perspectives on the value of American Society of Clinical Oncology clinical guidelines as reported by oncologists and health maintenance organizations.
Guidelines can help create a safer medical system by keeping physicians abreast of current evidence and standards of care. However, when guidelines contain outdated information or fail to adequately provide updates, then adherence to guidelines may create a more dangerous medical system. A study by Shekelle et al
reviewing guidelines from the 1990s found that at least 10% of guideline recommendations were no longer valid after 3.6 years and that 50% were no longer valid by 5.8 years.
In the present study, the mean age of the interventional practice guidelines was 5.2 years. During this time, significant recommendations have become outdated. Since publication of the 2009 ASGE guideline “Management of Antithrombotic Agents for Endoscopic Procedures,”
an additional 3 antithrombotic agents, dabigatran etexilate, rivaroxaban, and apixaban, with unique mechanisms of action have been Food and Drug Administration approved.
Following the recommendations for warfarin would potentially cause harm to a patient. Likewise, the ASGE guideline “A Consensus Document for Bowel Preparation Before Colonoscopy” lists aqueous sodium phosphate as an acceptable bowel preparation.
American Society of Colon and Rectal Surgeons; American Society for Gastrointestinal Endoscopy; Society of American Gastrointestinal and Endoscopic Surgeons A consensus document on bowel preparation before colonoscopy: prepared by a task force from the American Society of Colon and Rectal Surgeons (ASCRS), the American Society for Gastrointestinal Endoscopy (ASGE), and the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES).
This product was removed from the market in 2008 owing to phosphate-induced nephropathy. Although the guideline contains an addendum noting a Food and Drug Administration alert regarding this product, there is no addendum noting that this product was withdrawn. Recently, 2 bowel-cleansing preparations have been approved, Suprep (Braintree Laboratories Inc) (approved in 2010) and Prepopik (Ferring Pharmaceuticals) (approved in 2012),
but neither are included in the guidelines. Issues were also found in the SCAI guidelines. For example, the peripheral arterial disease guideline from 2005 was updated in 2011 with some recommendations no longer being advised.
ACC/AHA 2005 guidelines for the management of patients with peripheral arterial disease (lower extremity, renal, mesenteric, and abdominal aortic): a collaborative report from the American Association for Vascular Surgery/Society for Vascular Surgery, Society for Cardiovascular Angiography and Interventions, Society for Vascular Medicine and Biology, Society of Interventional Radiology, and the ACC/AHA Task Force on Practice Guidelines (Writing Committee to Develop Guidelines for the Management of Patients With Peripheral Arterial Disease).
2011 ACCF/AHA focused update of the guideline for the management of patients with peripheral artery disease (updating the 2005 guideline): a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines.
However, there is no mention in the 2005 guideline indicating that an update is present, with recommendation changes such as, “endovascular repair of infrarenal aortic and/or common iliac aneurysms is reasonable in patients at high risk of complications from open operations because of cardiopulmonary or other associated diseases” deleted in the 2011 update because endovascular repair can be considered even in good surgical candidates.
ACC/AHA 2005 guidelines for the management of patients with peripheral arterial disease (lower extremity, renal, mesenteric, and abdominal aortic): a collaborative report from the American Association for Vascular Surgery/Society for Vascular Surgery, Society for Cardiovascular Angiography and Interventions, Society for Vascular Medicine and Biology, Society of Interventional Radiology, and the ACC/AHA Task Force on Practice Guidelines (Writing Committee to Develop Guidelines for the Management of Patients With Peripheral Arterial Disease).
2011 ACCF/AHA focused update of the guideline for the management of patients with peripheral artery disease (updating the 2005 guideline): a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines.
Especially in the area of procedures, where the risk of harm is potentially higher, current evidence and recommendations are crucial to making correct decisions.
In addition, standardization of the methods used to grade the evidence and formatting of the documents are necessary. Clinicians should not have to assess the utility of the recommendation based on differing methods for grading evidence. None of the interventional medicine subspecialties used a standard method for grading evidence or for presenting the recommendations. One method that allows the clinician to quickly assess the strength of the recommendation and the potential likelihood that the evidence supporting the recommendation may change over time is the GRADE (Grading of Recommendations Assessment, Development, and Evaluation) system.
This system grades the quality of evidence as high, moderate, low, and very low. High quality indicates that further research is unlikely to change the estimated effect. Moderate, low, and very low quality portends variable levels of diagnostic certainty within which it is possible or likely that future studies may alter current recommendations.
This system allows the physician to have a clear and quick understanding of the strength of the recommendation that he or she is providing to his or her patients. Ideally, a uniform grading system and a standardized layout for practice guidelines should be established.
Finally, as the IOM notes, practice guidelines “lie at the intersection of medical research, education, and practice.”
When guidelines fail to grade the evidence supporting the recommendations and base clinical recommendations on expert opinion, the validity of the recommendations is questionable. Previously, Mendelson et al
reported that COIs in authors of practice guidelines may result in distrust of the guidelines and may lessen their value in establishing standards of care. We found that 62% of the interventional societal guidelines failed to comment on COIs and when reported, nearly all had substantial COIs. Jones et al
Clear policies are needed during the guideline development process to mitigate this issue. Although many societies have their own COI policies, a uniform policy is lacking. The Council of Medical Specialty Societies, however, has developed several policies limiting COIs in the development of practice guidelines. At each stage of development of the guidelines there are several reviewers and voters involved to minimize potential biases.
Without a uniform system providing unequivocal transparency, practitioners using practice guidelines may have difficulty evaluating which guidelines have potentially biased recommendations.
This study had several limitations. Multiple systems were used to grade the quality of evidence by the various societies. To provide adequate analysis of the level of evidence supporting the recommendations, the different systems were merged into 1 system (ABC). However, all the systems could not be merged effectively. To limit potential bias, combining the evidence into a single grading system was performed by 2 of us. In addition, only the guidelines published on the societal websites were reviewed. Guidelines published by other societies may exist but were not included.
Conclusion
Most medicine interventional practice guideline recommendations are rooted in low-quality evidence. Also, most guidelines fail to disclose any potential COIs. When present, numerous potential COIs were noted. Furthermore, guidelines were not updated frequently and contained outdated recommendations. A more uniform and standardized process is necessary for the development and maintenance of practice guidelines across all societies. Given the potential issues raised in this study, clinicians should be prudent regarding the use of current societal guideline recommendations.
Prevalence of financial conflicts of interest among panel members producing clinical practice guidelines in Canada and United States: cross sectional study.
Perspectives on the value of American Society of Clinical Oncology clinical guidelines as reported by oncologists and health maintenance organizations.
American Society of Colon and Rectal Surgeons; American Society for Gastrointestinal Endoscopy; Society of American Gastrointestinal and Endoscopic Surgeons
A consensus document on bowel preparation before colonoscopy: prepared by a task force from the American Society of Colon and Rectal Surgeons (ASCRS), the American Society for Gastrointestinal Endoscopy (ASGE), and the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES).
ACC/AHA 2005 guidelines for the management of patients with peripheral arterial disease (lower extremity, renal, mesenteric, and abdominal aortic): a collaborative report from the American Association for Vascular Surgery/Society for Vascular Surgery, Society for Cardiovascular Angiography and Interventions, Society for Vascular Medicine and Biology, Society of Interventional Radiology, and the ACC/AHA Task Force on Practice Guidelines (Writing Committee to Develop Guidelines for the Management of Patients With Peripheral Arterial Disease).
2011 ACCF/AHA focused update of the guideline for the management of patients with peripheral artery disease (updating the 2005 guideline): a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines.
Potential Competing Interests: Dr Leffler has served as a consultant for or has received grants from Prometheus Laboratories, Alba Therapeutics, Alvine Pharmaceuticals, and Shire. Dr Cheifetz has served as a consultant for or has received grants from Abbott Laboratories, Janssen Pharmaceuticals, Warner Chilcott, Given Imaging, Prometheus Laboratories, and Pfizer.
In their article published in the January 2014 issue of Mayo Clinic Proceedings, Feuerstein et al1 reported that only a relatively small number of the guidelines from interventional subspecialty societies are based on evidence derived from randomized controlled trials. This scenario is not limited to guidelines developed by interventional medicine societies; guidelines developed by other professional organizations are also largely based on low levels of evidence.2-4 This factor is largely related to lack of research focused toward bridging gaps in the evidence base.
00872-0&id=gr1.jpg){kind=link}
00872-0&id=gr2a.jpg){kind=link}
00872-0&id=gr2b.jpg){kind=link}