Embed Size (px)
Citation preview
Using Program Evaluation to Drive Outcomes &
Funding for Sustainability
Andrea M. Jensen, CHESEnvironmental Health Educator
• Participants will understand components of a strategic evaluation plan and individual evaluation plan
• Participants will understand how evaluation can be used for quality improvement and to demonstrate outcomes
• Participants will learn specific examples of how evaluation was used in the Utah Asthma Program to improve quality of asthma interventions and demonstrate program results
Evaluation critical for:• Quality improvement
o Look critically and objectively at program processeso Identify strengths and weaknesseso Make course corrections as necessary
• Demonstrating outcomes o Look objectively at program outcomeso Determine if programs are meeting their purposeo Make course corrections as necessary
Providing measurable outcomes to a funder builds the case for continued
funding
“Conversations for Advancing Action” Podcast
Series
• “Success builds success and if people understand that you are building something successfully and you’re doing good things and you can articulate where your successes are, and how you can achieve a way to build on those challenges. Everybody wants to bet on a winner and that’s the way to show it to them.”
• Maureen Wilce, Air Pollution and Respiratory Health Branch at CDC’s National Center for Environmental Health
We will discuss:
• How to develop a strategic & individual evaluation plan
• How the Utah Asthma Program used evaluation
to:o Improve quality of asthma interventionso Demonstrate program outcomes
Utah Asthma Program
Established in 2002
Utah County Asthma Program
• 13 LHD’s
• 2 funded LHD’s
• Large state o (8 hrs x 5 hrs)o Many rural areas
http://freepages.genealogy.rootsweb.ancestry.com/~tolleygenealogy/utahmap.html
Backgrounds?• Government?
• Nonprofit?
• Nurses?
• Respiratory therapists?
• Health Educators? (Public Health)
Personal interest
Interior Designer Health Educator
12 Hospitalizations
Previous work plan
More projects
More projects
Asthma listserv
CDC’s Framework for Evaluation
• Can be used for:
o Programs/Interventions
o Partnerships/Coalitions
o Surveillance systems
o Communication effortso Anything else you want
to evaluate
http://www.cdc.gov/asthma/program_eval/guide.htm
Introduction: State Asthma Program Evaluation Guide
Module 1: Evaluation and Your State Asthma Program
Module 2: Implementing Evaluations
Module 3: Evaluating Partnerships
Module 4: Evaluating Asthma Surveillance
Module 5: Evaluation of Services and Health Systems Interventions
Module 6: Economic Evaluation for State Asthma Programs
OverviewStrategic Evaluation
PlanIndividual Evaluation
Plan
• Overarching plan that shows evaluation strategies over a period of time
• Provides an overview and timeline of evaluations to be conducted
• Detailed plan for conducting a specific evaluation
Strategic Evaluation Plan (SEP)
• Explains how a strategic approach to evaluation will assist your program in meeting its purpose & goals
• Components include:o Program background and purpose of SEPo Methods for developing and updating the SEPo Proposed evaluations
• Make a prioritized list• Include a summary of each evaluation’s purpose• Include a timeline for conducting the evaluations
o Communication plan• How will information on the SEP results be shared?
Individual evaluation plan
• What are you going to evaluate?
• How are you going to evaluate it?
Components of an individual evaluation plan
• Evaluation purpose
• Description of what you are evaluatingo Not how you will evaluate, but what you are evaluating
• e.g. a certain intervention, partnership efforts, communication efforts, etc.
Components of an individual evaluation plan,
cont.
• Evaluation designo Stakeholder needso Evaluation questions
• What 3-5 major questions do you intend to answer through this evaluation?
o Description and rationale for evaluation design/methods
• Gather credible evidenceo Outline data collection methods for each evaluation
question
CDC Evaluation Checklist
http://www.cdc.gov/asthma/program_eval/AssessingEvaluationQuestionChecklist.pdf
Components of an individual evaluation plan,
cont.• Data analysis and interpretation
o Indicators and standardso What constitutes success?o Who will be involved in drawing and justifying conclusions?
• Use and communication of evaluation findingso How will evaluation findings be used? By whom?o How will evaluations findings be disseminated?
• Evaluation managemento Identify the evaluation teamo Define roles and responsibilities for everyone on the team
Using Evaluation for Quality Improvement:
Recess Guidance• Identify if the Recess
Guidance is being used by schools
• Are schools implementing it correctly?
• Identify strengths & weaknesses
• Make it more effective
Partnership Evaluation: Local Health Departments
(LHDs)• Identify strengths &
weaknesses
• How to improve the LHD grant process and required grant content for future LHD grant endeavors
Surveillance Systems Evaluation
• Critical review of the asthma surveillance system
• Assess strengths & weaknesses
• Make recommendations for system improvement
• Focus special interest in simplicity, flexibility, data quality, and usefulness of the system
Communication Evaluation: Asthma
Listserv
• Understand why people take part in the Utah Asthma Program Listserv
• What changes would make it more useful to participants?
Asthma Listserv Evaluation
• Evaluation Questions:o What are the barriers to partner using listserv for sharing
information?o What improvements should be made to increase
communication on the listserv (should it become a more traditional listserv)?
o What types of messages sent on the listserv are most useful?o Why are members involved in the listserv?
• Data sourceso Literature reviewo Listserv membership excel sheet and recent listserv messageso Online questionnaire sent to listserv participants
Asthma Listserv Evaluation - Recommendations
• Continue the listserv in the current format of 1-way• Email is the preferred communication medium
• Prefer 1-4 messages per month • Having large gaps between emails is ineffective • Find ways to increase the number of people who
follow links/look at materials outside of the basic email message
Process and Outcome Evaluation: Asthma
Telehealth• Online asthma education program that offers CE
to nurses and respiratory therapists
• Two part evaluation o Process evaluation in 2011o Outcome evaluation in 2014
Asthma Telehealth Process Evaluation,
2011• The purpose of this evaluation was to identify
course corrections for the Asthma Telehealth Program. The evaluation looked at fidelity of implementation, reach, and dose delivered
• Recommendations to improve the program were implemented over a period of two years
• Implementation of evaluation results paved the way for a future outcome evaluation
Recommendations• Timing
• Continue quarterly sessions (as opposed to monthly)
• Presentation content and delivery:
• Work with presenters on presentation skills
• Include slides with takeaway points and a call to action to reinforce key messages
• Summarize content half way through • Enhance the question-and- answer section following the
webinar
Recommendations
• Topics• Topic selection must include criteria from EPR-3 guidelines• Use healthcare provider focus groups to identify relevant
topics
• CE and Assessments• Expand CE credit to physicians and pharmacists• Collect data on participant change or patient level for
outcome evaluation• Evaluate skill/ability change based on web-based education
Using Evaluation to Demonstrate Outcomes: Asthma Telehealth Series
• Outcome Evaluation – 2014
o Identify effectiveness of educating health providers on NAEPP guidelines-based care (in order to improve treatment and care for people with asthma)
o Identify how well the Telehealth program has succeeded in achieving its ultimate goals
Evaluation DesignEvaluation Questions• Why do some participate in only one webinar?• What changes have participants made? • How has learning increased? • Is the right audience staying engaged?• How are topics keeping professionals on the forefront
of information for their profession?
Data Collection• Registration forms, pre- and post-test data, and
evaluation sheets completed by participants• Google form email survey 2 weeks after program
participation to assess practice changes
Outcome Evaluation Results
• More RNs than RTs participated
• Increase in participants registering for more than one program over the last 3 years o (30.1% – 43.7%)
• Process evaluation worked to increase numbers!
70% showed improvement from pre-post test
Behavior Change Resulting from Telehealth
Participation
• 50% intended to make a practice change
• 50% selected ‘confirmed my current practice’
• 57% of reported implemented the call to action practice changeo This was a much higher percentage than the criteria
for success (set at 35%)
Recommendations:• Continue with the call to action for participants
• Link the learning objectives of the program to skill-level measureso Participants would indicate his or her skill level before and
after
• Add instant messenger feed
• Promote Telehealth sessions to rural providers
New funding - 2014• Comprehensive Asthma Control through
Evidence-based Strategies and Public Health - Health Care Collaboration grant
• Started over with new 5 year strategic plan……
School-based Education
Home-based Education
Linkages to care• Prescription drug
assistance
• Home remediation
• Insurance navigator
• Referral to free/low cost clinics
Already Evaluating!
Planned Evaluations• System Scan for Expansion of Asthma Services in
Target Areas • GHHI and Asthma Home Interventions • Coverage and Reimbursement for Asthma
Services • Community Health Workers (CHWs) Asthma
Training & Utilization of CHWs • Asthma Feedback and Referral Mechanism • Asthma Business Case Utilization • Care Coordination Mechanism and Partnerships
Questions?Andrea M. Jensen, CHES
Environmental Health EducatorUtah County Health Department
