Embed Size (px)
Citation preview
4/7/17
1
Effective Program Planning and Evaluation for Public
Health Professionals Presented by the
Nevada Public Health Training Center
Julie Lucero, Ph.D. MPH, Assistant Professor, University of Nevada, Reno, School of Community Health Sciences
Richelle O’Driscoll, M.A., Faculty Emerita, University of Nevada, Reno, Division of Health Sciences
Schools
Community Centers
Employers
Transit
Elected Officials
Doctors
EMS
Law Enforcement
Nursing Homes
Fire
Corrections
Mental Health
Faith Institutions
Civic Groups
Non-Profit Organizations
Neighborhood Organizations
Laboratories
Home Health
CHCs
Hospitals
Tribal Health
Drug Treatment
Public Health Agency
Public Health System
The Essential Public Health Services 1. Monitor health status
2. Diagnose and investigate health problems
3. Inform and educate
4. Mobilize communities to address health problems
5. Develop policies and plans
6. Enforce laws and regulations
7. Link people to needed health services
8. Assure a competent health services workforce
9. Evaluate health services
10. Conduct research for new innovations
4/7/17
2
Group Activity
u Please share your name, the department, and the program that you are affiliated with
u Please put a sticky on the essential service that best describes your area
The Essential Public Health Services 1. Monitor health status
2. Diagnose and investigate health problems
3. Inform and educate
4. Mobilize communities to address health problems
5. Develop policies and plans
6. Enforce laws and regulations
7. Link people to needed health services
8. Assure a competent health services workforce
9. Evaluate health services
10. Conduct research for new innovations
Survey Says…..
u Findings u Competencies u Training Objectives
4/7/17
3
Training Objectives
u Participants will learn how to participate in public health program planning processes related to your job.
u Participants will learn how to identify ways to monitor and evaluate programs for their effectiveness and quality.
u Participants will be given resources, templates, and tools to help them to plan and evaluate programs founded in quality improvement and sustainability planning.
Generalized Model
Needs Assessment u “ a planned process that identifies the reported needs of
an individual or group” Gilmore (2012)
u “process by which health education specialists gather information regarding health needs and desires of a population” (NCHEC, SOPHE, & AAHE, 2010)
u “process of determining, analyzing, and prioritizing needs, and in turn, identifying and implementing solution strategies to resolve high priority needs” Wilkin (2000)
4/7/17
4
Key Concepts: u Determining the priority population u Identifying and evaluating needs—describing
“problems” of a target population and possible solutions to these problems
u Needs: u A gap between “what is” and “what should be” u A gap between real and ideal.
u Determining the degree to which the needs are being met u If needs are not being met, is there also a need to enhance
capacity to do so?
Three-Phase Model of Needs Assessment
Explore “What Is”
• What do we already know about the population group or the health issue?
• What data sources are you utilizing?
• What are the need indicators?
• What are the preliminary priorities?
Gather & Analyze Data
• Determine target group • Collect data to determine
the current state of target group in relation to desired outcome
• Formulate need statement, discrepancy between “what is” and “what should be”
• Prioritize need, based on data
Make Decisions
• Set Priority Needs • Identify Possible Solutions • Select Solutions • Propose an action plan
(program) • Include evaluation metrics
Steps in Conducting a Needs Assessment
Step 1 Step 2 Step 3 Step 4 Step 5 Step 6
Determining the purpose and scope
Gathering data
Analyzing data
Identifying risk factors linked to health problems
Identifying the program focus
Validating the need
4/7/17
5
Acquiring Needs Assessment Data
u Primary: Data you collect yourself
u Surveys, Community Forums, Meetings, Focus Groups, Observations, Self-Assessments, Written Questionnaires, Fact to Face Interview,
u Secondary: Data already collected by someone else and available for your use
u Collected from governmental agencies, nongovernmental agencies, existing records, literature. internet
Example: Obesity prevention program
u Purpose: to develop an intervention concentrating on improving healthy weight and nutrition by improving knowledge of health, media, and financial literacies among American Indian and Hispanic families.
u Working with three non-profit organizations
u Site data: US census, town demographics, and community resources
u Service data from organizations
u Literature threaded from:
u Hispanic women & children and obesity
u American Indian women & children and obesity
u Influence of improved literacies on weight
Group Activity- Needs Assessment
u Identify a method that you might utilize to collect primary data for your health issue.
u Identify two sources you would consult to collect secondary date for your health issue.
u Share your findings with the class.
4/7/17
6
Generalized Model
u Concise statement describing the purpose of the program.
u A.k.a. program overview or program aim
u Can reflect grounding philosophy of program
u Components of a mission statement:
u Primary functions activities of the program. Most important functions/operations/offerings
u Purpose of the program. Primary reasons for doing the program.
u Stakeholders, who will benefit/participate in the program
4/7/17
7
+
u Structure of a mission statement:
The mission of [program name] is to [programs primary purpose] by providing [programs primary function/activities] to [program stakeholders]. Add clarifying statements if needed.
OR
The __________(program) will _________ for ________ by ___________.
Checklist for Mission Statement
ü Is the statement clear and concise?
ü Is the statement distinctive and memorable?
ü Does the statement clearly state the purpose of the program?
ü Does the statement indicate the primary function or activities of the program?
ü Does the statement indicate who the stakeholders are?
ü Does it reflect the organizations priorities and values?
u Statements of what the program intends to accomplish.
u Broad general statements of what the program intends to accomplish.
u Goals should provide a framework for determining the more specific educational objectives of a program
u Provide overarching expectations.
u Should be consistent with the mission of the program and the mission of the institution.
4/7/17
8
Structure of a goal statement To (action verb) (object) (modifiers)
To create goals, measurable objectives, related activities, and expected outcomes for a public health program
To develop advisory board to oversee program activities.
To increase the participants knowledge of the history of Halloween.
u Objectives are brief, clear statements that describe steps of programs that accomplish goals.
u In measureable terms, objectives specify the changes your program sets out to make.
u “Constructing objectives for a program is the most important skill for planners.” (McKenzie, Neiger, & Thackeray, 2013, p. 144)
Specific and Measurable Objectives
u Outcome to be achieved (reduce, increase, evaluate, etc.)
u Condition = when (after the session, with 2 days of,…)
u Criterion = measurement (10%, 53 people, …)
u Who or what will change?
u Who is going to do what, when and to what extent?
4/7/17
9
Objectives Types
u Process Objectives: day-to-day activities that are needed to accomplish the goal.
u Impact Objectives: immediate observable effects of a program.
u Learning Objectives
u Change in awareness, knowledge, attitude and/or skills
u Behavioral Objectives
u Change in behavior
u Environmental Objectives
u Non-behavior causes of health (social/physical/political/economic)
u Outcome Objectives:
Example: Obesity prevention program
Goal
u Develop and train a Community Research Team (CRT) in two sites that serve Spanish-speaking and American Indian Families.
u Objective
u Within 3 months, the non-profits will recruit 5 women from each site to comprise the CRT.
Group Activity- Goals and Objectives
u Develop two goals for your health topic.
u Develop two SMART objective for each goal (total of four).
u Share your findings with the class.
4/7/17
10
Generalized Model
Multidirectional
Comm
unication Model
Health Communication Strategies u The five traditional channels include:
1. Intrapersonal u Health care, health coaches, hotlines
2. Interpersonal u Small classes, support groups
3. Organizational
u Church bulletins, company or agency newsletters
4. Mass media
u Newspaper, billboards, magazines, PSAs, and more
5. Social media u User or consumer generated, organized, and distributed
u Information can be revised or updated almost immediately
u Typically low cost in terms of creation and maintenance
u Facebook, blogs, LinkedIn, Twitter, text messaging
4/7/17
11
© 2013 Pearson Education, Inc.
Health Policy / Enforcement Strategies
u Six phases of policy making (Block, 2008)
1. Agenda setting
2. Policy formulation
3. Policy adoption
4. Policy implementation
5. Policy assessment
6. Policy modification
Health Promotion Intervention: what to consider
Designing Appropriate Interventions
u Key Terms–
u Best practices – using interventions that have undergone critical review of multiple research & evaluation studies (Green & Kreuter, 2005)
u Best experiences – fall short of best practices but show promise
u Best processes – original interventions based on theory
u Segmenting – dividing a broader population into smaller groups
u Tailoring – intervention created for specific needs, interests, & circumstances
4/7/17
12
Adapting a Health Promotion Intervention
Implementing the intervention “The act of converting planning, goals, and objectives into action through administrative structure, policy, management activities, procedures, regulations, and organizational actions of new programs” (Timmerick, 1997)
“Setting up, managing, and executing of a project” (Keyser, 1997)
Example: Obesity prevention program
Goal
u Develop and train a Community Research Team (CRT) in two sites that serve Spanish-speaking and American Indian Families.
u Objective
u Within 3 months, the non-profits will recruit 5 women from each site to comprise the CRT.
u Activity
u Once a woman with knowledge of the community and culture is identified, the program coordinator will inform her of the program and assess her interest in becoming a CRT member.
4/7/17
13
Table 2. Family Wisdom Project Aims & Associated Activities
(Per Fiscal Year in Quarters) Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q1 Q2 Q3 Q4Apr-June July-Sept Oct-Dec Jan-Mar Apr-June July-Sept Oct-Dec Apr-June July-Sept Oct-Dec Jan-Mar
IRB application approval XRecruit and hire CRT X X XCRT meetings X X X X X X X X X X XReview community resources & health communication materials X X X
Provide training sessions to CRT X X X X X X X XPre-post survey and focus groups of CRT member peer trainers X X X
Develop sustainability plan and prepare for R01 submission X X X X
Recruit participants XPre and post training survey (see table 1 for specifics) X X X X X X X X
Implement training sessions (see table 1 for specifics) X X X X X X X X X
Conduct focus group with participants (see table 1 for specifics) X X X
Data analysis & interpretation of survey, FG & training session notes X X X X X X X X
Review & refine training sessions (a.k.a.curriculum) based on quantitative and qualitative data.
X X X X
Recruit CoP X X X XCoP meetings X X X X X X X X X X X X XDevelop print materials X X X X X X XConduct focus group with community members to test print materials X
FG analysis & print material revision X X XDisseminate print materials and offer trainings X X X X X X X X
X
X
X
YR01 YR02 YR03
Q4Jan-Mar
Aim One: Develop and train a Community Research Team (CRT) in two sites that serve Spanish-speaking and American Indian Families.
X
Aim 3. Develop and Disseminate community education trainings locally and nationally.
X
X
X
Aim 2. Pilot test community-training sessions and explore environmental, family and individual elements that contribute to healthy weight and nutrition within families in years 2 and 3.
X
Barriers to Implementation Tasks and timelines
4/7/17
14
Group Activity- Developing and implementing an intervention
u Identify the type of intervention you would like to implement for your health topic (e.g. communication, policy, health promotion, etc.). u You may need to revise your goals.
u Develop two activities for each SMART objective. u Develop timeline u Develop a budget u Share your findings with the class.
Generalized Model
Program Evaluation u What is evaluation?
u Purposeful, systematic collection of information
u Documents the effectiveness and impacts of program
u Establishes accountability
u Identifies areas needing change and improvement
4/7/17
15
Purpose of Evaluation
u General intent u To gain insight u To improve how things are done u To determine what the effects of the program are u To affect those who participate in it u Accountability u Contribute to scientific base for interventions u Inform policy decisions
Process Evaluation (formative)
u Helps you understand the relationship between specific program elements and program outcomes
u Evaluates the in extent to which a program is being implemented as planned (Harris, 2010,p. 207)
u Identifies the key components of an intervention that are effective (including for whom and under what conditions)
Types: Formative (Process and implementation)
u Conducted while the program is ongoing u Is the program being implemented as it was designed?
u Can include information from pretesting and pilot
u Do the students or clients understand the program’s concepts?
u What are the misconceptions about the program?
u Are all program implementers implementing the program in the same way?
“Program Evaluation Model 9-Step Process”, Janet E. Wall. Sage Solutions, [email protected]
4/7/17
16
Types: Formative (Process and implementation)
u Is the program being implemented on schedule?
u Is there sufficient time to implement all aspects of the program?
u What aspects of the program do not seem to be working as well as you intended?
u Do program implementers need additional training on the program?
u Are there any negative outcomes surfacing?
“Program Evaluation Model 9-Step Process”, Janet E. Wall. Sage Solutions, [email protected]
Types: Summative (Outcome) u Post-Program
u What did the program accomplish?
u Did the program reach its goals and objectives?
u What impact did the program have on its recipients?
u What were the outcomes?
u Who benefited from the program?
u How much was the benefit?
“Program Evaluation Model 9-Step Process”, Janet E. Wall. Sage Solutions, [email protected]
Types: Summative (Effectiveness)
u Was the benefit greater with this program as compared with another program?
u Did all types of clients benefit from the program?
u What were the positive outcomes?
u What were the negative outcomes?
u What should be improved/changed in the program?
u Does the benefit of the program warrant the cost?
“Program Evaluation Model 9-Step Process”, Janet E. Wall. Sage Solutions, [email protected]
4/7/17
17
Impact Evaluation
u Focuses on “the immediate observable effects of a program, leading to the intended outcomes
u Is associated with behavioral impact or change (Windsor, et.al, 2004)
u Measures of awareness, knowledge, attitudes, skills and behaviors
Outcome Evaluation (summative)
u Focuses on an ultimate goal or product of a program or treatment
u Generally measured in the health field by mortality or morbidity data in a population, vital measures, symptoms, signs or physiological indicators on individuals (Green & Lewis, 1986, p. 364)
u Basically, it makes a determination of the effect of a program or policy on its beneficiaries (Harris, 2010)
Basic Process for Evaluation u Define the Purpose and Scope of the Evaluation u Specify the Evaluation Questions What Do You Want to
Know? u Specify the Evaluation Design u Create the Data Collection Action Plan u Collect Data u Analyze Data u Document Findings u Disseminate Findings u Feedback to Program Improvement
4/7/17
18
KU Work Group for Community Health and Development. (2015). Chapter 36, Section 1: A Framework for Program Evaluation: A Gateway to Tools. Lawrence, KS: University of Kansas. Retrieved March 30, 2017 from the Community Tool Box: http://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/framework-for-evaluation/main
What to collect, measure and analyze
u The program's capacity to deliver services u The participation rate u The level of client satisfaction u The amount of intervention exposure (how many
people were exposed to the program, and for how long they were exposed)
u Changes in participant behavior u Changes in community conditions or norms u Changes in the environment (e.g., new programs,
policies, or practices) u Longer-term changes in population health status
(e.g., estimated teen pregnancy rate in the county)
Results and Recommendations
u Evaluation needs to answer specific questions.
u Answers to the questions need to be gathered and analyzed.
u Summarize and translate the information into synthesize into patterns and trends.
u Interpret the findings based on the data (evidence).
u Judgements are statements about the merit, worth, or significance of the program.
u Recommendations are actions to consider as a result of the evaluation
4/7/17
19
QI Process
Program Evaluation-Summary
u What is the best way to evaluate?
u What are we learning from the evaluation?
u How will we use what we learn to become more effective?
Group Activity- Program Evaluation
u Develop a question for each that could be used to evaluate the program for your health topic: u process evaluation
u impact evaluation
u outcome evaluation.
u What collection method would you utilize with each?
4/7/17
20
Assess
Plan
Implement Evaluate
Disseminate
