Evaluation of the National Guideline Clearinghouse™

AHRQ Annual ConferenceSeptember 2011

Acknowledgements

Thank-you to: Survey

respondents Focus group

participants Key

informants AMIA and

ACCP AHRQ

Mary Nix Judi Consalvo Gov Delivery

Staff Marjorie Shofer

HIT Portfolio

AMA and AHIP ECRI Institute NGC

EvaluationPET Florence Chang Belinda Ireland Richard Shiffman Katrin Uhlig Cally Vinz

Evaluation Project Team Members

AFYA, Inc.• Michelle Tregear• Jenice James• Debra Dekker• Craig Dearfield• Ajay Bhardwaj• Robin Pugh-Yi

The Lewin Group• Carol Simon• Jaclyn Marshall• Jacob Epstein

Evaluation Objectives

Gain a better understanding of how NGC: Is used by its stakeholders (including

AWARENESS among key stakeholders) Supports dissemination of evidence-based

clinical practice guidelines and related documents

Has influenced efforts in guideline development, implementation, and use

Can be improved

Data Collection Methods

Quantitative Data Web-based Survey

▪ Skip logic and Branching▪ Respondents solicited by

e-mail lists for AHRQ, AMA, and AHIP

Qualitative Data Focus groups (4)

▪ Stakeholder-specific Key informant

interviews (26)▪ Mix of stakeholders

Mixed-Methods Approach

Key Project Milestones• CIPP evaluation framework – Logic

model to develop key questions• Developed instruments which were

Informed by the PET• Received OMB clearance Feb 2011• Conducted survey, focus groups, and

interviews (Mar – Jul 2011)

NGC objectives

DATA & Inputs: DisseminationActions:

PDA downloads NGC update

service weekly email

NGC electronic discussion list forum

RSS Feeds HTML

Process Outcomes:

Web Hits Subscribers Multiple

subscriptions Guidelines

published annually

Percent of submitted guidelines accepted

Research Requests

Time from abstraction to verification

Intermediate Outcomes:

Awareness Frequency of use Length of use Reasons for use Saves time Level of trust Satisfaction New guideline

development Reference for

quality efforts Influence

clinical practice Input into

decision support systems

Disseminate through Web 2.0

Reference for research

Longer-term Outcomes:

Improve health care quality

Promote efficiency

Reduce health care costs

Increase scientific knowledge to improve decision-making

External Factors:1) Availability and quality of other guideline sites, including medical societies, government entities, and commercial products2) Funding allocated to the development, implementation, and dissemination of guidelines3) Barriers to implementing guidelines4) Clinicians attitudes towards guidelines

feedback

Guidelines Guideline

comparisons Guideline

syntheses Expert

commentary Annotated

bibliography Recommended

readings FDA alerts NGC browse Detailed search

Resource to obtain objective, detailed information on guidelines and to further their dissemination, implementation, and use

NGC objectives

DATA & Inputs: DisseminationActions:

PDA downloads NGC update

service weekly email

NGC electronic discussion list forum

RSS Feeds HTML

Process Outcomes:

Web Hits Subscribers Multiple

subscriptions Guidelines

published annually

Percent of submitted guidelines accepted

Research Requests

Time from abstraction to verification

Intermediate Outcomes:

Awareness Frequency of use Length of use Reasons for use Saves time Level of trust Satisfaction New guideline

development Reference for

quality efforts Influence

clinical practice Input into

decision support systems

Disseminate through Web 2.0

Reference for research

Longer-term Outcomes:

Improve health care quality

Promote efficiency

Reduce health care costs

Increase scientific knowledge to improve decision-making

External Factors:1) Availability and quality of other guideline sites, including medical societies, government entities, and commercial products2) Funding allocated to the development, implementation, and dissemination of guidelines3) Barriers to implementing guidelines4) Clinicians attitudes towards guidelines

feedback

Guidelines Guideline

comparisons Guideline

syntheses Expert

commentary Annotated

bibliography Recommended

readings FDA alerts NGC browse Detailed search

Resource to obtain objective, detailed information on guidelines and to further their dissemination, implementation, and use

Survey Sample Characteristics

Referral Source

Source Count

Total 9,389

• AHRQ 9,298 (99%)

• AHIP 42

• AMA 49

Respondent Demographics

Survey Reach Majority from the U.S.

(87%)

Occupation Mix 56% - Providers, clinicians,

nurses 13% - Researchers,

librarians, or similar 12% - Consultants,

managers, administrators 19% - Other

Survey SampleMajority familiar with

AHRQ (99%)

Survey Respondent CountsSection / Module

Count

Total 9,389 (100%)NGC Unaware 2,075 (22.1%)

NGC Aware 7,314 (77.9%)

Non NGC User 1,395 (19.3%)

NGC User 5,828 (80.7%)• Guideline Developer 1,076 (18.5%)

• Health Provider 3,271 (56.1%)

• Medical Librarian 204 (3.5%)

• Informaticians 292 (5.0%)

• Researcher 1,219 (20.9%)

• Policymaker 1,219 (20.9%)

• Measure Developer 351 (6.0%)

Stakeholder-specific questions

NGC questions

NGC awareness and use, demographics, other guideline source questions

NGC Awareness, Use, and Other Guideline Sources

Results – NGC Awareness

Key Notes• Majority of respondents

derived from AHRQ e-mail invitation

• Awareness of NGC greatest among AHRQ and AHIP respondents

AHRQ Opportunity to increase physician awareness of NGC

• Awareness of NGC in 2011 substantially higher than in 2001 evaluation

2001 20110%

20%

40%

60%

80%

100%

30%

79%70%

21%

AwareUnaware

Awareness Over Time

AHRQ AHIP AMA0%

20%40%60%80%

100%78% 83%

46%

Awareness by Survey Source

N=9,289 N=42

N=49

Results – Guideline Sources Used

Survey Findings Most use multiple

sources to find CPGs (3-5 most common)▪ PubMed, government

sources, general search engines, medical/professional societies

Qualitative Findings: NGC often cited as

“first go-to source”

Key Finding

Most NGC users equally satisfied

or more satisfied with NGC

compared to other guideline

sources

Much less satisfied

Slightly less satisfied

About equal

Slightly more satisfied

Much more satisfied

No ResponseN=5,828

Results – Degree to which NGC Fulfills Users’ Specific Needs when Using NGC

Determining Coverage of Services

Controlling Healthcare Expenditures

Reducing Errors/Malpractice

**Comparing CPGs

Class Assignments

*Developing quality measures

*Research

*Developing CPGs

*Clinical Decisionmaking

**Knowledge Building

**Finding CPGs

To a Great Extent Some-what Very Little Not at all

Key Finding

NGC is

doing well on

these needs

“indicate the degree to which use of the NGC Web site fulfills your needs for:”

Results – NGC Compared to Other Sources in Meeting Needs for Guidelines

Comparing CPGs

Finding CPGs

Developing Quality Measures

Reducing Errors/Malpractice

Clinical Decisionmaking

Developing CPGs

Controlling Healthcare Expenditures

Professional Knowledge Building

Academic/Medical Research

Class Assignments

Determining Coverage of Services

17%

16%

20%

17%

18%

23%

19%

17%

21%

18%

24%

52%

57%

55%

58%

58%

54%

58%

61%

57%

60%

55%

31%

27%

25%

24%

24%

23%

23%

23%

22%

22%

21%

Other Source Better About Equal NGC BetterPercent of users

FindingNGC does

betteror about equal

compared to other sources

This is supported

by qualitative

findings

NGC – Trustworthiness and Inclusion Criteria

Results – Trustworthiness

Survey Findings 75% -very good or

good▪ 80% - would definitely,

very likely, or probably recommend NGC to colleague

No differences by key stakeholder group

Qualitative Findings:Differences by key stakeholder group▪ Guideline developers and informaticians less trusting ▪ Others note that they “trust” the content because “it

comes from AHRQ”

Very poor

Poor

Acceptable

GoodVery good

Don’t knowNo Response

0% 25% 50% 75%

N=5,828

Results – NGC Inclusion Criteria

Survey Findings 63% - appropriate

▪ By length-of-use:longer users more likely to find the criteria too loose

No differences by keystakeholder group

Qualitative Findings:Differences by key stakeholder group▪ Guideline developers/informatics specialists said they are

‘too loose’▪ Policymakers, medical librarians, researchers generally

satisfied

Stringent

Appropriate

Loose

Don't know

No response

0% 20% 40% 60% 80%

N=5,828

Results – NGC Inclusion Criteria

Qualitative Findings (Guideline Developers and Informaticians):

▪ “…the quality of the criteria was good when NGC started out but it has gotten more complicated...”

▪ “…If the goal is to be ‘all inclusive,’ then the criteria are fine. [but]….there needs to be some other ways to separate the wheat from the chaff.”

▪ “They could raise the bar.”

▪ “I’d say too loose…. I think there’s a belief that: A) NGC creates these guidelines…and then, B) there’s also a belief that NGC somehow has a very rigorous process for only allowing certain guidelines in, or certain types of very high quality guidelines, or that it’s an endorsement of these guidelines. And it isn’t.” AHRQ Opportunity to revisit NGC’s Inclusion Criteria

Results – NGC Age Criterion

AHRQ Opportunity to revisit NGC’s Age Criterion

Survey Findings ~Equal # say 5 years

is “appropriate” or “too long”

▪ Reduce to 3 years most common selection

No differences by keystakeholder group

Qualitative Findings: Common theme: the age criterion is too long

Too long Appropriate Too short Don’t know Missing0

10

20

30

40

50

4 yrs 3 yrs 2 yrs0%

20%

40%

60%

Stakeholder-SpecificNGC Influences

NGC Influences – StakeholdersKey Take-Aways from Survey Findings

Respondents, by group, indicated NGC greatly influenced: Guideline developers’ ability to identify guidelines, and

develop and use quality measures

Health providers’ ongoing learning activities, clinical decision-making processes, and ability to identify guidelines

Medical librarians’ ability to meet their client’s needs

Medical librarians’ and researchers’ ability to identify current and high quality guidelines

Measure developers’ measure development activities and approach to identifying guidelines

Policymakers’ and purchasers’ ability to identify guidelines and convert clinical information

NGC Influences – StakeholdersKey Take-Aways from Survey Findings

Respondents, by group, indicated NGC greatly influenced: Guideline developers’ ability to identify guidelines, and

develop and use quality measures

Health providers’ ongoing learning activities, clinical decision-making processes, and ability to identify guidelines

Medical librarians’ ability to meet their client’s needs

Medical librarians’ and researchers’ ability to identify current and high quality guidelines

Measure developers’ measure development activities and approach to identifying guidelines

Policymakers’ and purchasers’ ability to identify guidelines and convert clinical information

NGC Influences – Guideline Developers Participating in NGC

Survey Findings (n=199) NGC guideline submitters reported greater NGC

influence for guideline updating frequency, and how organizations document or report their guidelines

Qualitative Findings (n=24) NGC serves primarily as a source for locating

guidelines NGC’s age criterion applies some “pressure” to stay

current NGC has had little influence on how guideline

developers do their work– e.g., methodology, reporting

AHRQ Opportunity to increase knowledge among guideline developers about how to create and report trustworthy guidelines

Noteworthy finding 65% said

excellent or good

21% were neutral

14% said fair or poor

Poor

Fair

Neutral

Good

Excellent

0% 10% 20% 30% 40% 50%

5%

9%

21%

41%

24%

N=138

Question: How would you rate NGC's dissemination of your organization’s guidelines?

AHRQ Opportunity to identify additional efforts to enhance the dissemination of guidelines

NGC Influence – Dissemination of CPG’s Survey Findings for Participating Developers

NGC Enhancements

Likely use of Potential EnhancementsSurvey Findings

Enhancements Two thirds said they would “definitely,”

“very likely,” or “probably” use the following NGC enhancements if available:• Ratings of guideline quality• Subject-specific e-mail alerts

Enhancements for providers 72% - having NGC content at

point-of-care would be useful 66% - would take CME if available

Enhancements for informaticians 52% - NGC Summaries as XML file

according to the Guideline Elements Model (GEM) or other similar

Examples of Potential Enhancements

• Ratings of guideline quality

• Subject-specific e-mail alerts

• Access to archived guidelines

• Additional data download options and xml formats

AHRQ Opportunity to invest in major enhancements that will increase the value of NGC

Additional Enhancements – Common Themes Qualitative Findings

Additional Common Themes Guideline Developers

Commentary/responses to guidelines from users

Guideline developer conferences / methodology workshops

Informatics Specialist Assessment of the executability

of guidelines

Medical Librarians Integration with other Web sites

(PubMed) Quicker access to “new”

guidelines

Researchers Assessment of attributes in IOM

standards for guidelines More information about treatment

in multi-morbidity or comorbid conditions

Conclusions and Recommendations

Key Take-Aways for Expanding NGC’s Use and Impact

AHRQ Opportunities include:Build on NGC’s user base, including with health providers

Revisit the NGC inclusion criteria

Revisit the guideline age criterion

Increase knowledge among guideline developers about how to create and report trustworthy guidelines

Enhance guideline dissemination efforts

Invest in major enhancements to the NGC Website that will provide significant added value