The Endocrine Society Clinical Practice Guidelines:A Self-Assessment

Robert A. Vigersky, Shalender Bhasin, and Kathryn A. Martin

Walter Reed National Military Medical Center (R.A.V.), Bethesda, Maryland 20889; Brigham andWomens Hospital (S.B.), Boston, Massachusetts 02115; and Massachusetts General Hospital (K.A.M.),Boston, Massachusetts 02114

In 2005, after many years of discussion about whetheror not The Endocrine Society (TES) should produceClinical Practice Guidelines (CPGs), TES Council ap-proved a carefully designed program to produce 2 to 4high-quality CPGs each year. TES has published 20CPGs to date (1), and several more are expected to bepublished in the next 2 years. Since its inception, theCPG program has been unique in many ways. First, ahighly regulated pathway on the route to publication(Figure 1) permits the selection, review, and vetting ofCPGs at multiple levels, including a final peer and ed-itorial review at the Journal of Clinical Endocrinologyand Metabolism (JCEM). Second, the guidelines havebeen selected and produced without support from phar-maceutical or device-manufacturing companies. Third,all CPG expert panels have included at least 1 interna-tional member. Fourth, all CPGs have been translatedinto patient versions. Fifth, the CPGs are routinely re-viewed and updated every 3 years. Finally, TES adoptedthe Grades of Recommendation Assessment, Develop-ment and Evaluation (GRADE) approach for evaluatingthe evidence and producing recommendations (2).

The GRADE Working Group began as an interna-tional effort to resolve confusion about how to best andmost clearly rate evidence and express recommenda-tions. Many governmental organizations and profes-sional societies use the GRADE approach, including theAgency forHealthcare Research andQuality, the Amer-ican College of Physicians, and UpToDate in the UnitedStates; theNational Institute for Clinical Excellence andthe BritishMedical Journal in the United Kingdom; andthe World Health Organization (3). Although there are

multiple systems of grading evidence, the GRADEmethod is the only one that rates the strength of therecommendation (Strong or Weak) based on a balanceof benefits vs harms, the confidence in the magnitude ofestimated effect on an outcome, and the weighing ofpatient values and preferences. Although this methodallows for Strong recommendations to be made withlow (L) or very low (VL) levels of evidence and viceversa, this should not be a frequent occurrence. TESengaged the Knowledge and Evaluation Research(KER) Unit at the Mayo Clinic to provide methodolog-ical assistance to each of the CPG expert panels, whoseexpertise generally is in clinical medicine and not inmethodology or epidemiology.

In this issue of JCEM, members of the KERUnit haveundertaken a unique exerciseassessing the appropri-ateness of 357 recommendations contained in all 17CPGs that were published by TES between 2006 and2011 (4). Their major finding was that 206 of the 357recommendations were Strong, of which 121 werebased on L/VL levels of evidence. Of the 121, 53 werein categories such as sensible alternatives do not existor recommend additional research. KER Unit mem-bers then retrospectively applied a recently publishedschema by theGRADEWorkingGroup of paradigmaticsituations where Strong recommendations with L/VLevidence might be appropriate (5). Using this new tax-onomy of paradigmatic situations to determine the ap-propriateness of the remaining 68 Strong recommenda-tions that were based on L/VL levels of evidence, theyfound that it was appropriately applied to 35 recom-mendations and inappropriately applied to 33 (616).

ISSN Print 0021-972X ISSN Online 1945-7197Printed in U.S.A.Copyright 2013 by The Endocrine SocietyReceived May 20, 2013. Accepted May 22, 2013.

For article see page 3246

Abbreviations: CPG, Clinical Practice Guideline; GRADE, Grades of RecommendationAssessment, Development and Evaluation; KER, Knowledge and Evaluation Research; L,low; VL, very low.

S P E C I A L F E A T U R E

E d i t o r i a l

3174 jcem.endojournals.org J Clin Endocrinol Metab, August 2013, 98(8):31743177 doi: 10.1210/jc.2013-2300

Supplemental Table 1 (published on The Endocrine So-cietys Journals Online web site at http://jcem.endojournals.org) shows those deemed to be inappropriate (personalcommunication from Dr Juan Pablo Brito).

Anumber of important questions arise as a result of thisself-assessment. First, howwell is theCPGprogramofTESdoing? It would appear that the CPG program is doingextremely well overall. Brito et al (4) assert that 27% (33of the 121 Strong recommendations with L/VL evidence)of the recommendations were unjustified. However, itmay be more informative to look at these 33 recommen-dations more globally using either 357 (the total numberof recommendations) or 306 (357 53 sensible alter-natives or required additional research recommenda-tions) as the base. Thus, the percentage of unjustifiedrecommendations is 9 or 11%, respectively. Another ap-proach is to assess the number of unjustified recom-mendations in each guideline. For example, in 2008, therewere 5 guidelines published with a total of 106 recom-mendations, of which 12% were unjustified accordingto the new GRADE schema (5 of 22 in The Diagnosis of

Cushings Syndrome; 3 of 27 fromPrimary Prevention of Cardiovas-cular Disease and Type 2Diabetes inPatients at Metabolic Risk; 4 of 12from Case Detection, Diagnosis,and Treatment of Patients with Pri-mary Aldosteronism; 0 of 17 fromEvaluation and Treatment of Hir-sutism in Premenopausal Women;and 1 of 38 from Prevention andTreatment of Pediatric Obesity).The diversity in the frequencyof unjustified recommendationsmay be due to the fact that certainexpert panels had more assistancefrom themethodologist than others,that for some CPGs it is inherentlymore difficult to develop clear rec-ommendations, or that the evalua-tion by the GRADE WorkingGroups new schema was un-equally applied. It is likely thatthere are elements of all 3 explana-tions underlying the findings ofBrito et al (4).

Second,doestheanalysisbyBritoetal (4) diminish the validity or the clin-ical importanceof those33Strongrec-ommendations based on L/VL levelsof evidence? At the very worst, theserecommendations represent expertopinion that, in the absence of high-

quality evidence, can provide useful and in some instancesthe only feasible form of guidance for making difficult clin-ical decisions by cutting through the cacophony of conflict-ing, weak, or nonexistent data. Many of the 33 recommen-dations based on L/VL levels of evidence either addressedsafety concerns or were related to the interpretation orapplication of laboratory tests. In addition, it should benoted that few laboratory tests have undergone extensivetesting in large randomized trials. Historically, there hasbeen a paucity of population-based, rigorously derivedreference ranges for most hormones and analytes. Al-though TES has recently initiated efforts to generate andharmonize population-based reference ranges for severalhormones, it remains the case that the thresholds for de-fining hormone deficiency or hormone excess for mostendocrine disorders are necessarily based on either a L/VLlevel of evidence or expert opinion.

Third, do systematic reviews help expert panelsmakethe correct recommendations? Brito et al (4) expressconcern that strong recommendations are made with

1. Clinical Guideline Sub-commiee selects topic and expert panel chair.

2. Sub-commiee reviews conict-of interest statements of panel and approves/disapproves members

3. Endocrine Society Council approves topic and panel

4. Panel meets to review evidence and begin draing the guideline . Expert panel formulates up to 2 meta-analyc quesons.

5. KER Unit methodologist and panel review results of meta-analysis and incorporate them into dra guideline.

6. Methodologist assists panel chair with wording of recommendaon and grading of evidence.

7. Concurrent review of guideline by Clinical Guidelines Sub-commiee, Clinical Aairs Core Commiee, Society member (on line), co-sponsoring organizaons (if applicable)

8. Taskforce reviews, responds, and revises dra

9. Council Reviews and Approves

10. Manuscript submied to JCEM for peer-review

11. Panel responds to peer-review and nal manuscript is published

Figure 1. The Endocrine Societys process of Clinical Practice Guideline Development.

doi: 10.1210/jc.2013-2300 jcem.endojournals.org 3175

evidence yielding limited confidence in the estimates ofeffect about the available options. It might be expectedthat systematic reviews would ameliorate this legiti-mate concern. However, Brito et al (4) found that sys-tematic reviews were used in 14 of 125 (12%) Strongrecommendations that were made with L/VL quality ofevidence. Weak recommendations were made withL/VL quality of evidence to a similar degree (21 of 122,or 19%). Thus, it is unclear whether using more sys-tematic reviews, which are often strongly considered inevaluating the evidence, would be helpful in avoidinginappropriate recommendations.

Fourth, how does TES compare with other organi-zations? It would appear that TES compares quite fa-vorably with other organizations in the quality and va-lidity of CPGs. For example, it is reassuring that the rateat which CPGs of TES apply Strong recommendationswith L/VL evidence is similar to the American Associ-ation of Blood Banks and the Society for Vascular Sur-gery, which made Strong recommendations with L/VLevidence 64 and 65%of the time, respectively, using theGRADE system, although their appropriateness wasnot addressed (17, 18). Indeed, this may be viewed as aninherent flaw of the GRADE approach or in its appro-priate application. Comparing guideline developmentof TES to that of other endocrinology organizations isdifficult because of the differences in methodologiesused. No other endocrinology professional society usesthe GRADE approach, and the criteria for developmentand approval of guidelines of other societies are lessrigorous and/or less transparent (19, 20).

Fifth, how can TES improve the quality of its guide-lines? Quality improvement is a goal of any organiza-tion. Indeed, the KER Units effort is an important anduseful attempt to assess and improve on their own per-formance as well as to determine how the GRADE systemcan best be used to make clinically valid recommenda-tions. TES whole-heartedly embraces the evolving na-ture of the GRADE guideline development process andis committed to closely integrating methodological ad-vice into every guideline. By recognizing paradigmaticsituations that better define when Strong recommenda-tions with L/VL evidence are appropriate, TES shouldbe able to continue to be in the vanguard of producingthe highest quality guidelines for our members and theirpatients. Evidence-based practice involves complex de-cision-making that is based not only on the availableevidence but also on patient characteristics and prefer-ences, values of the patient and physician, and ever-changing uncertainties. It is certain that high-qualityevidence will not be available in all clinical situations,and in situations where high-quality evidence is not

available, expert opinion and careful synthesis of low-quality evidence will continue to appropriately guideclinical practice.

Acknowledgments

Address all correspondence and requests for reprints to:Robert A. Vigersky, MD, Walter Reed National MilitaryMedical Center, Endocrinology and Diabetes Service, 8901Wisconsin Avenue, Bethesda, Maryland 20889. E-mail:robert.vigersky@us.army.mil.

Disclosure Summary: R.A.V., S.B., and K.A.M. are the first,second, and current chairs of theClinical PracticeGuideline Sub-committee, respectively. The opinions expressed in this paperreflect the personal viewsof the authors andnot the official viewsof the United States Army or the Department of Defense.

References

1. The Endocrine Society. Current Clinical Practice Guidelines Website. http://www.endocrine.org/endocrine-press/clinical-practice-guidelines. Accessed June 9, 2013.

2. Atkins D, Best D, Briss PA, et al; GRADEWorking Group.Gradingquality of evidence and strength of recommendations. BMJ. 2004;328:1490.

3. GRADEWorking Group.Organizations that have endorsed or thatare using GRADE Web site. http://www.gradeworkinggroup.org/society/index.htm. Accessed May 14, 2013.

4. Brito JP, Domecq JP, Murad MH, Guyatt GH, Montori VM. TheEndocrine Society Guidelines: when the confidence cart goes beforethe evidence horse? J Clin Endocrinol Metab 2013;98:32463252.

5. Andrews JC, Schunemann JH,OxmanAD, et al.GRADEguidelines15: going from evidence to recommendationdeterminants of arecommendations direction and strength. J Clin Epidemiol. 2013;66(7):726735.

6. Melmed S, Casanueva FF, Hoffman AR, et al. Diagnosis and treat-ment of hyperprolactinemia: an Endocrine Society clinical practiceguideline. J Clin Endocrinol Metab. 2011;96:273288.

7. AbalovichM, AminoN, Barbour LA, et al.Management of thyroiddysfunction during pregnancy and postpartum: an Endocrine Soci-ety clinical practice guideline. J Clin Endocrinol Metab. 2007;92:S1S47.

8. Bhasin S, CunninghamGR, Hayes FJ, et al. Testosterone therapy inmenwithandrogendeficiency syndromes: anEndocrine Society clin-ical practice guideline. J Clin Endocrinol Metab. 2010;95:25362559.

9. Funder JW, Carey RM, Fardella C, et al. Case detection, diagnosis,and treatment of patientswith primary aldosteronism: anEndocrineSociety clinical practice guideline. J Clin Endocrinol Metab. 2008;93:32663281.

10. Speiser PW, Azziz R, Baskin LS, et al. Congenital adrenal hyper-plasia due to steroid 21-hydroxylase deficiency: an Endocrine So-ciety clinical practice guideline. J Clin Endocrinol Metab. 2010;95:41334160.

11. Nieman LK, Biller BM, Findling JW, et al. The diagnosis of Cush-ings syndrome: an Endocrine Society clinical practice guideline.J Clin Endocrinol Metab. 2008;93:15261540.

12. HembreeWC, Cohen-Kettenis P, Delemarre-van deWaal HA, et al.Endocrine treatment of transsexual persons: an Endocrine Societyclinical practice guideline. JClinEndocrinolMetab. 2009;94:31323154.

3176 Vigersky et al Clinical Practice Guidelines: A Self-Assessment J Clin Endocrinol Metab, August 2013, 98(8):31743177

13. Freda PU,BeckersAM,KatznelsonL, et al.Pituitary incidentaloma:an Endocrine Society clinical practice guideline. J Clin EndocrinolMetab. 2011;96:894904.

14. Rosenzweig JL, Ferrannini E, Grundy SM, et al. Primary preventionof cardiovascular disease and type2diabetes in patients atmetabolicrisk: an Endocrine Society clinical practice guideline. J Clin Endo-crinol Metab. 2008;93:36713689.

15. August GP, Caprio S, Fennoy I, et al. Prevention and treatment ofpediatric obesity: an endocrine society clinical practice guidelinebased on expert opinion. J Clin Endocrinol Metab. 2008;93:45764599.

16. Molitch ME, Clemmons DR, Malozowski S, et al. Evaluation andtreatmentof adult growthhormonedeficiency: anEndocrine Society

clinical practice guideline. JClinEndocrinolMetab. 2006;91:16211634.

17. Djulbegovic B, Trikalinos TA, Roback J, ChenR,Guyatt G. Impactof quality of evidence on the strength of recommendations: an em-pirical study. BMC Health Serv Res. 2009;9:120.

18. Murad MH, Montori VM, Sidawy AN, et al. Guideline methodol-ogyof the Society forVascular Surgery including the experiencewiththe GRADE framework. J Vasc Surg. 2011;53:13751380.

19. BennettWL,OdelolaOA,Wilson LM, et al.Evaluation of guidelinerecommendations on oral medication for type 2 diabetesmellitus. Asystematic review. Ann Intern Med. 2012;156:2736.

20. VigerskyRA.A review and critical analysis of professional societiesguidelines for pharmacologic management of type 2 diabetes mel-litus. Curr Diab Rep. 2012;12:246254.

Download The Endocrine Societys multi-journal, full-text app to stay up-to-date on all your mobile devices.

Available at Apple App Store https://itunes.apple.com/us/app/endo-pubs/id438308412?mt=8 Available at Android Market https://play.google.com/store/apps/details?id=com.sheridan.tes

doi: 10.1210/jc.2013-2300 jcem.endojournals.org 3177