Upload
trantram
View
216
Download
0
Embed Size (px)
Citation preview
Evaluating Collective Impact
Community Systems Group, LLC. 1
Community Systems Group, LLC 2
Should I Stay?
1. What does it mean to establish shared measures?
• For the our coalition’s capacity and process?
• For our coalition’s accomplishments (the dose)?
• For our coalition’s outcomes?
2. How do we connect these measures to show
collective impact?
3. What software is available to support data collection
and analysis.
Goals
Community Systems Group, LLC 3
1. What does it mean to establish shared measures?
• For the our coalition’s capacity and process?
Goals
4
Evidence-Based Processes
Community Systems Group, LLC 4
5
Lit Review: Quality / Degree
Community Systems Group, LLC 5
6
Behavior vs.
Perception
Community Systems Group, LLC 6
7
Benchmark
Community Systems Group, LLC 7
1. What is our shared (and evidence-based) definition of capacity?
2. What are our agreed upon measures?
3. How could having common measures of capacity benefit our work?
4. Would if be beneficial if all coalitions in our community used the same measures?
Community Systems Group, LLC 8
Critical Reflection:
Community Systems Group, LLC 9
1. What does it mean to establish shared measures?
• For the our coalition’s capacity and process?
• For our coalition’s accomplishments (the dose)?
Goals
Presenting Question:
What is it that partnerships can do that their
members cannot do (or do as well) alone?
Community Systems Group, LLC 10
1. Community Change: Instances
New or improved policies, programs and practices facilitated by the initiative in the community.
Definitions
Community Systems Group, LLC 11
12
Falmouth Prevention Partnership
• History of the Partnership
• Relationship with CSG
Community Systems Group, LLC 12
13
Community Changes are new or modified programs, policies and
practices facilitated by the partnership to reduce substance abuse in Falmouth. This includes modifications to the physical environment.
Examples: • For the first time a Social Host Lesson at Falmouth High School was implemented as part
of the health curriculum for students in the 9th through the 12th grade. 375 students attended the social host laws class. (New Program: May 1, 2012)
• The Falmouth Police Department dispatching added a new category to track liquor law
violations for underage persons. Prior to this there was not a specific way to collect data on underage liquor law violations. (New Practice: September 13, 2011)
• The FPP, Falmouth Public Schools and the Falmouth Police Department co-sponsored and required attendance at a Social Host Pre Prom Assembly for the parents/caretakers of children attending the Junior Prom. (New Program: May 9, 2012)
13 13 Community Systems Group, LLC 13
14 14 Community Systems Group, LLC 14
The Falmouth Prevention Partnership facilitated:
64 Community changes intended to reduce underage alcohol use.
42 Community changes intended to build opportunities for pro-social involvement or that targeted all drug use.
30 Community changes intended to reduce Rx abuse.
28 Community changes to increase community capacity (such as new partnerships).
164 Documented new or modified programs, policies, and practices facilitated by the Partnership from 2009 to 2013 in order to reduce substance abuse in Falmouth, MA.
15
Dimensions of Intensity2
Dimension Scoring Rationale
Duration High Medium Low
Ongoing More than Once One Time
Emerging evidence and expert consensus suggest that community changes that are sustained are more likely to affect behavior change and population-level outcomes.
Reach High Medium Low
≥ 21% of population = 6% to 20% of population ≤ 5% of population
Emerging evidence and expert consensus suggests that exposure to community changes is important to affect outcomes.
Strategy High Medium Low
Policy, Consequences, Access Social Support Information, Skills
Emerging evidence and expert consensus suggests that community changes addressing policy and access are more likely to achieve outcomes.
2 Adapted from Collie-Akers, V.L., Fawcett, S.B., and Schultz, J.A. (2013). Measuring progress of collaborative action in a community health effort. Rev Panam Salud Publica, 34(6): 422-428.
15 15 Community Systems Group, LLC 15
Helping ou create a healthier communities Helping you create healthier communities 16
Dimensions of Intensity2
Community Change Duration Reach Strategy Score
Launch of the sticker shock campaign in 18 liquor retail stores with 10, 000 stickers.
Medium Medium Low 1.20
Pre Prom Assembly which is mandatory for all parents of students going to the Junior prom.
Medium Medium High 2.10
For the first time participated in "Credit for Life" Community Service Booth.
Low Low Low 0.30
2 Adapted from Collie-Akers, V.L., Fawcett, S.B., and Schultz, J.A. (2013). Measuring progress of collaborative action in a community health effort. Rev Panam Salud Publica, 34(6): 422-428.
Helping ou create a healthier communities 17 n = 15
i = 15.75 n = 16
i = 23.70 n = 36
i = 38.70 n = 26
i = 37.95 n = 32
i = 45.15 n = number of changes i = total intensity score
Intensity of Changes to Reduce Underage Alcohol Use facilitated by the
Falmouth Prevention Partnership
0
5
10
15
20
25
30
35
40
45
50
2009 2010 2011 2012 2013
Provide Information
Provide Support
Build Skills
Change the Physical Design
Access & Barriers
Policy Change
Rewards & Consequences
sum
an
nu
al in
ten
sity
sco
re
Year (First 3 Quarters)
1. Community Change: Instances
New or improved policies, programs and practices facilitated by the initiative in the community.
2. Services Provided: Contact Hours
Events that give information, build skills or provide social support for people in the community.
Definitions
Community Systems Group, LLC 18
Helping ou create a healthier communities Community Systems Group, LLC 19
Services Provided are events that impart information, develop skills,
and give social support such as through classes, screenings, alternative activities, and workshops. Records on services provided include the number of classes conducted, participants, length, and location.
Examples: • The Falmouth Prevention Partnership provided an in-service training for K-12
Adjustment and Guidance Counselors in Falmouth Public Schools on use of evidenced-based prevention strategies.
• FPP participated in a "Credit for Life" Community Service Booth. The event is for high school seniors to prepare them for financial literacy and community volunteerism.
• The Alcohol and Marijuana Action Team brought the "It's All About YOU" presentation by Richard Campbell to the Junior and Senior Falmouth High School students in a one-hour assembly format.
20
Helping ou create a healthier communities Helping you create healthier communities 20
0
200
400
600
800
1000
1200
1400
1600
1800
2009 2010 2011 2012 2013*
Tota
l C
on
tact
Ho
urs
FPP: Services Provided 2009-2013
Provide SocialSupport
ProvideInformation
Build Skills
3212 youth & 3120 adults received services facilitated by the Partnership.
*2013 data are for the first 9 months of the year.
Collecting Output Data
3. Media: Impressions
Coverage of the initiative, its projects or issues in any media channel such as through newspapers, newsletters, on-line, radio, and television.
Definitions
Community Systems Group, LLC 21
Helping ou create a healthier communities Community Systems Group, LLC 22
Media are instances of coverage of the initiative, its projects, or issues
in the newspaper, on-line (“new media”), radio, television, and other channels. Records on media include the number of impressions and the media channel.
Examples:
• "Program Aims To Educate Parents Who Serve Alcohol“ article in Falmouth Enterprise describes how parents of high school students will receive a letter in the mail warning them of the legal consequences of allowing teenagers to drink alcohol in their homes.
• "Parents Who Host Lose The Most" public service announcement ran 55 times from Dec.1 -Dec.24, 2011 on local public access television, Channel 13.
• Dr. Mike Bihari, FPP Chair, wrote the Friday, September 27, 2013 “Risky Business” column in the Falmouth Enterprise. The article contained statistics on underage drinking and the correlation between teen perception of parental disapproval and a decrease in underage drinking.
Helping ou create a healthier communities Community Systems Group, LLC 23
0
200
400
600
800
1000
1200
2009 2010 2011 2012 2013*
Tota
l Im
pre
ssio
ns
in T
ho
usa
nd
s
FPP: Media Impressions 2009-2013
Risky Business column
Community Education
Sticker Shock
Parents Who Host Losethe Most
*2013 data are for the first 9 months of the year.
Helping ou create a healthier communities Community Systems Group, LLC 24
“Parents Who Host Lose the Most” Campaign
Average Impressions per Falmouth Resident
2011 2012 2013
6.11 22.32 9.88
Collecting Output Data
3. Media: Impressions
Coverage of the initiative, its projects or issues in any media channel such as through newspapers, newsletters, on-line, radio, and television.
4. Resources Generated: Dollar Value
Acquisition of funding for the initiative and the community through grants, donations or gifts in-kind.
(usually cash, materials or time)
Definitions
Community Systems Group, LLC 25
1. Catalyst for needed change.
Community Change
2. Assure essential supports.
Services Provided
3. Foster an ongoing community conversation.
Media
4. Generate and focus resources.
Resources Generated
The Four Metrics
Community Systems Group, LLC 26
1. What is our shared (and evidence-based) definition of accomplishments /outputs?
2. What are our agreed upon measures?
3. How could having common measures of outputs benefit our work?
4. Would if be beneficial if all coalitions in our community used the same measures?
Community Systems Group, LLC 27
Critical Reflection:
Community Systems Group, LLC 28
1. What does it mean to establish shared measures?
• For the our coalition’s capacity and process?
• For our coalition’s accomplishments (the dose)?
• For our coalition’s outcomes?
Goals
Community Systems Group, LLC 29
1. What does it mean to establish shared measures?
• For the our coalition’s capacity and process?
• For our coalition’s accomplishments (the dose)?
• For our coalition’s outcomes?
2. How do we connect these measures to show
collective impact?
Goals
30
Analysis of Contribution
1. Collect output data (measure the dose).
2. Establish a time sequence.
3. Demonstrate a plausible mechanism.
4. Account for alternative explanations.
5. Show similar effects in similar contexts.
Community Systems Group, LLC 30
31
Establishing a Time Sequence
Method
• Document when the intervention took place.
(e.g. the four outputs of change, service, media and resources)
• Document when the outcomes took place.
(e.g. local conditions, risk factors, behaviors, health consequences, etc.)
• Visually plot these to establish a time sequence.
Community Systems Group, LLC 31
32
Falmouth implements a survey of students modeled on the Communities That Care instrument (PRIDE Survey).
Measures the incidence and prevalence of substance use, delinquency, and related problem behaviors and the risk and protective factors that predict those problems in communities.
Surveys were conducted in 2009, 2011, and 2013.
Students in 8th through 12th grades were surveyed (In 2009 students in 7th grade were also included).
Establishing a Time Sequence
Community Systems Group, LLC 32
33 i = 15.75 i = 23.70 i = 38.70 i = 37.50 i = 42.60
Relationship in Time Between Changes to Reduce Underage Alcohol Use and
Self-Reported Rates of Alcohol Use in the Past 30 Days by 11th Graders
0
5
10
15
20
25
30
35
40
45
50
2009 2010 2011 2012 2013
Sum Intensity
Score
sum
an
nu
al in
ten
sity
sco
re
100
90
80
70 60 50
40
30
20
10
0
% of 11th Grade Students Reporting Alcohol Use in the Past 30 Days.
% = 58.30 % = 56.50 % = 33.50
Helping ou create a healthier communities Helping you create healthier communities 34
0
10
20
30
40
50
60
70
80
90
100
7th 8th 9th 10th 11th 12th
2009
2010
2011
2012
2013
2014
2015
2016
2017
Change occurs around high school with Classes 2014, 2015, and 2016. (p = .05)
Class of 2017 is showing change starting in the 8th grade. (p = .05)
Past 30 Day Use of Alcohol by Class Cohort
Baseline Data
Intervention Data
35
• There has to be a relationship in time in order for
there to be a contribution.
• However, a relationship in time does not prove a
contribution.
• If improvements happen after the “dose” then an
exploration is appropriate.
Helping ou create a healthier communities Helping you create healthier communities 35
Establishing a Time Sequence
Most Common Error
Assuming a causal relationship exists just because one thing follows another.
ex post facto error
Superstitious sports fans who believe their lucky hat helped the team win the game.
Coalitions that see 30 day use of alcohol go down and suggest they are the cause.
Establishing a Time Sequence
36 Helping ou create a healthier communities Helping you create healthier communities 36
37
Analysis of Contribution
1. Collect output data (measure the dose).
2. Establish a time sequence.
3. Demonstrate a plausible mechanism.
Community Systems Group, LLC 37
38
Mechanism:
“the means by which the intervention
could have achieved intended effects.”
Mechanism of Effect
38 Helping ou create a healthier communities Helping you create healthier communities 38
39
Mechanism of Effect
39 Helping ou create a healthier communities Helping you create healthier communities 39
By what means could the changes, services, and media implemented by the Partnership have affected rates of underage alcohol use?
1. Facilitated changes, services, and media implemented interventions with an existing evidence-base.
• Social Host Laws • Parents Who Host Lose the Most • Sticker Shock • Guiding Good Choices • Pro-Social Opportunities
40
Mechanism of Effect
40 Helping ou create a healthier communities Helping you create healthier communities 40
By what means could the changes, services, and media implemented by the Partnership have affected rates of underage alcohol use?
2. Facilitated changes, services, and media implemented a comprehensive set of behavior change strategies.
• Provide Information • Provide Social Support • Build Skills • Change Access / Barriers • Alter Rewards / Consequences • Change the Physical Design of the Environment • Change Rules & Policies
41
Mechanism of Effect
41 Helping ou create a healthier communities Helping you create healthier communities 41
By what means could the changes, services, and media implemented by the Partnership have affected rates of underage alcohol use?
3. Pathway (logic model)
Past 30 Day Alcohol Use
Availability
Retail
Home Pro-Social Supports
Parental Attitudes
42
Alcohol Use and Related Factors by Class Cohort
7.9
26.1
33.5
95.6
90.7
92.9
29.3
55.5 65.6
56 56 58
44 48
48
0
10
20
30
40
50
60
70
80
90
100
7th 9th 11th
Class of 2014
31.8
56.5
86.5
78.9
44.2
68.1
48 51
43
45
0
10
20
30
40
50
60
70
80
90
100
9th 11th
Class of 2012
Past 30-DayUse
ParentalDisapproval
Availability
Pro-social:School
Pro-social:Community
Baseline or “Pre” After Interventions or “Post”
Mechanism of Effect
42 42 Helping ou create a healthier communities Helping you create healthier communities 42
43
By what means could the changes, services, and media implemented by the Partnership have affected rates of underage alcohol use?
3. Pathway (logic model)
Past 30 Day Alcohol Use
Availability
Retail
Home Pro-Social Supports
Parental Attitudes
Mechanism of Effect
43 43 43 Helping ou create a healthier communities Helping you create healthier communities 43
44
Analysis of Contribution
1. Collect output data (measure the dose).
2. Establish a time sequence.
3. Demonstrate a plausible mechanism.
4. Account for alternative explanations.
Community Systems Group, LLC 44
45
What else could it have been?
In research this is called “threats to validity.”
45 Community Systems Group, LLC 45
Account for Alternatives
46 46 Community Systems Group, LLC 46
Account for Alternatives
What else could it have been? [validity threats]
1. History: Events that co-occur with the
intervention and might account for the
observed change in outcomes.
2. Maturation: Processes within respondents
that co-occur with the intervention and might
account for the observed change in
outcomes.
47
What else could it have been? [validity threats]
3. Testing: The effects of taking a test upon
the scores of a second or additional testing.
4. Instrumentation: Changes in calibration of
instruments or in observers or scorers can
produce changes in measured results.
Account for Alternatives
47 47 Community Systems Group, LLC 47
48
Account for Alternatives
48 48 Community Systems Group, LLC 48
What else could it have been? [validity threats]
5. Regression: Groups or outcomes selected
for their extreme scores are likely to see
natural movement toward normal scores.
6. Selection: Differences between comparison
groups and intervention groups can bias
results.
49
Account for Alternatives
49 49 Community Systems Group, LLC 49
What else could it have been? [validity threats]
7. Mortality: Lost cases or cases with only
partial data can alter scores across time and
cause different results across groups.
50
A. Secular Trends
- National data from Monitoring the Future survey shows a downward trend in 30 day use of alcohol over the same time period.
- State data from the Youth Health Survey shows a downward trend in 30 day use of alcohol over the same time period.
+ Trends in Falmouth exceed those observed nationally or at the state level.
+ Secular trends not followed on other substances such as marijuana.
50 50 50 Community Systems Group, LLC 50
Account for Alternatives
51 51 51 51 Community Systems Group, LLC 51
Account for Alternatives
B. Concurrent Efforts
To better understand concurrent efforts in Falmouth key informant interviews were conducted with:
• State prevention staff • Local leaders in youth work and prevention
52 52 52 52 Community Systems Group, LLC 52
Account for Alternatives
B. Concurrent Efforts
State informants report no concurrent implementation of prevention efforts reaching Falmouth youth and families (one effort in Vineyard located elsewhere on the Cape).
Local informants reported three major efforts: • VIPS (focused on skills and mentoring) • Jack’s PACT • Together We Can (after-prom; recreation center)
FPP Partnered with all of these (funded, coordinated, co-implemented, helped to establish).
53 53 53 53 Community Systems Group, LLC 53
Account for Alternatives
B. Concurrent Efforts
State informants report no concurrent implementation of prevention efforts reaching Falmouth youth and families (one effort in Vineyard located elsewhere on the Cape).
Local informants reported three major efforts: • VIPS (focused on skills and mentoring) • Jack’s PACT • Together We Can (after-prom; recreation center)
FPP Partnered with all of these (funded, coordinated, co-implemented, helped to establish).
54
Conclusions
Influences on the national and state trends in rates of alcohol use may also be influencing rates in Falmouth.
VIPS, Jack’s Pact, and Together We Can are also making contributions to improved youth outcomes.
Much of these partner’s work is accounted for in the Partnership’s documented accomplishments / dose.
Additional dosage data from these community partners is required to quantify their potential contribution.
Account for Alternatives
54 54 54 54 Community Systems Group, LLC 54
55
Account for Alternatives
55 55 55 55 Community Systems Group, LLC 55
Conclusions
Only the Partnership was found to have implemented interventions consistent with the pathway effect observed.
National trends are inconsistent with the amount, timing, and specific nature of the observed effects (sudden, marked, and alcohol/parental attitudes specific).
56
Analysis of Contribution
1. Collect output data (measure the dose).
2. Establish a time sequence.
3. Demonstrate a plausible mechanism.
4. Account for alternative explanations.
5. Show similar effects in similar contexts.
Community Systems Group, LLC 56