Joan Twohey-Jacobs, Ph.D. Kristen Donovan, Ph.D.

August 12, 2014 Substance Use Disorders Statewide Conference

1

I. Prevention and Early Intervention (PEI) and the Mental Health Services Act (MHSA) II. Strategies to Engage Providers III. Measuring and Enhancing Collaboration IV. Overcoming Challenges Associated with Diverse Programs

2

3

!  Mental Health Services Act (MHSA) Proposition 63

!  Passed in 2004, became law in 2005 !  1% tax on personal income over $1 million

dollars !  Fund county and statewide mental health

services

4

Needs  Assessment  –  Quan.ta.ve  &  Qualita.ve  Data; Input  from  community,  Mental  Health  Board,  etc.

Stakeholder  Planning  Workgroups  –  Develop/assess  program/MHSA  

Component

Community  Leadership  CommiJee  –  Review  &  Approval

Behavioral  Health  Advisory  Board  –  30  Day  Pos.ng  for  Community  Review  &  Comment

County  Board  of  Supervisors  Approval

Behavioral  Health  Advisory  Board  

Approval

Public  Hearing-­‐  Facilitated  by  Behavioral  Health  Advisory  Board

Implementa.on

MHSA  Planning  &  Approval  Process

Policy/Priority Setting

5

!  Funds ear-marked for 5 areas: ◦  Community Services and Supports (CSS) ◦  Innovation (INN) ◦  Workforce Education and Training (WET) ◦  Capital Facilities and Technological Needs (CFTN) ◦  Prevention and Early Intervention (PEI)

6

!  20% of funds to be spent on PEI !  51% of PEI funds to 0-25 years old ◦  except small counties

!  Prevent mental illness from becoming severe and disabling

!  Improve timely access to people underserved by mental health system

!  “Help-first” instead of “Fail-first” !  Distinction between Prevention & Early

Intervention

7

!  Institute of Medicine Categories ◦  Prevention !  Universal !  Selective ◦  Early Intervention

8

9

Community  Needs  &  Reali.es  

Evidenced-­‐based  Prac.ces  Resources  

10

Engaging diverse providers

Measuring and enhancing collaboration

Challenges associated with diverse programs

A.  Explain process B.  Meet individually with providers C.  Train on tools D.  Present findings

11

!  Regular Provider Meetings !  Explain what is in it for them (WIIFT) !  Big Picture Strategies ◦  Engaging & useful gaphics ◦  To extent possible, keep it simple ◦  Meet the “system” where it is at in

order to enhance it

12

6. Analyze and report evaluation findings

5. Provide training and technical assistance to providers regarding evaluation

4. Develop countywide evaluation plan

3. Revise/fine existing and develop new tools as needed

2. Develop evaluation questions

1. Comprehensive assessment of data collection activities

13

What data are currently being

collected, how often, and from whom?

Meetings with PEI & QI Staff

Information Gathering

Meetings with Providers

14

Process Questions

• How is the program unfolding?

• What services are delivered?

• Who is participating? • Are the programs

implemented as planned?

Outcome Questions

• Are goals being met? • How do participants

benefit? • What changes in

attitudes, knowledge, skills, and/or behavior are occurring?

15

16

!  Training and TA ◦  Train to use tools ◦  Check on regularly

!  Analyze and report findings ◦  Final steps ◦  Meaningful for funder and provider

17

! With the data you collect, the department and the community-at-large can learn and benefit from the great work you are doing.

!  Participate in information gathering meetings with evaluator in assessment of tools phase; participate in trainings in preparation for evaluation data collection.

!  You can assure the collection of accurate and meaningful process/outcome data.

!  You are critical to the administration and collection of useful survey data.

18

!  Can take many formats ◦  By phone or in-person ◦  Broad or focused

!  Information gathering ◦  Typically 60-90 min.

!  Provide questions to interviewee !  Listen to concerns !  Build rapport !  Determine what will

work at the project/provider level

19

!  Slide presentation !  Administration Guidelines !  Allow time for questions !  Emphasize value of their role

20

o  Providers are critical to the administration and collection of useful survey data

o  Provider attitude toward the survey influences participants’ perceived value of their responses

o  The more consistent the data collection process across providers, the more valid the findings

21

o  Share/explain the purpose of the survey

o  Emphasize their input is important and valued

o  Be friendly, courteous, and appreciative

22

o Whenever possible, it’s important to ensure respondents know that their responses will remain anonymous o  No names are collected and no one will be able to

identify who provided the survey data

o  Don’t verbally administer the survey unless necessary; allow respondents to complete the survey on their own (unless they need assistance) o  Encourage honest responses o  Ensure that their responses will not affect receipt of

services o  Alleviate any concerns

23

24

 

 

o  Give respondents plenty of space to complete the survey to allow for anonymity

o  Direct the respondents to clearly mark their answers on the survey

o  Be familiar with all of the survey items so you are able to answer any questions from respondents

25

o  If the respondent is hesitant to answer a particular question, have him/her skip the question and move on to the next one

o  If a respondent is confused about an item,

tell him/her to answer the item to the best of his/her ability

o  Offer to complete the survey ONLY if the respondent appears to have special needs (reading is a challenge)  

26

o  Survey certain participants because you think they will produce better results

o  Provide any personal opinions or

comments regarding survey items

!  Demonstrates value of time and willingness to participate in data collection

!  Simple but powerful !  1-2 comments about each figure or finding !  Focus on positive findings ◦  Enhances motivation and maintains buy-in ◦  Allow for collaboration and discussion for

interesting findings

27

•  Very high satisfaction overall: 96% highly satisfied •  Leaders viewed as knowledgeable: 97% agreed •  Material viewed as useful for on-the-job: 98% agreed •  Participants would recommend the program: 97% agreed •  Early intervention participants rated treatment favorably:

9.5 avg. on 10-point scale

28

98%

89%

Prgm 2: Better able to recognize signs that someone maybe dealing

with a mental health problem or crisis

Prgm 1: More knowledgeable about mental health problems or

crises

% Agreed

30

87%

87%

92%

97%

96%

97%

92%

92%

89%

0% 50% 100%

I learned to avoid situations that might get me into

trouble with the law

Now, if someone treats me badly, I am more likely to

ignore him/her and walkaway rather than lash back

I see and understand the problems with getting involved with gangs

Group

Case Management

Seminar/Workshop

Lower than others

Lower than others

A.  Collective impact B.  PEI Network Measure C.  Collaboration Checklist D.  Provider Reflection Tool

31

◦  Assessing program/initiative effects is key to understanding what’s working, both individually at the program level, as well as collectively. ◦  COLLECTIVE IMPACT

5 Conditions Shared by Initiatives Achieving Large Scale Change through Collective Impact:

1.  Common Agenda 2.  Mutually Reinforcing Activities 3.  Backbone Support 4.  Continuous Communication 5.  Shared Measurement

Collec&ve  impact  is  not    just  a  fancy  name  for  collabora&on,  but  represents    a  fundamentally  different,    more  disciplined,  and  higher  performing  approach  to    achieving  large-­‐scale    social  impact.”    (Brown,  Kania  &  Kramer,  2012,  p.  2)

32

No  Interac+on  0  

Networking  1  

Coordina+on  2  

Collabora+on  3  

No  communica.on  with  organiza.on  outside  of  VCBH  Provider  Mee.ng  

Interacted  with  organiza.on;  loosely  defined  roles;  liJle  communica.on;  decisions    made  independently  

Share  informa.on;  some  defined  roles;  

frequent  communica.on;  some  shared  

decision  making  

Share  ideas  and  resources;  frequent  communica.on    and  mutual  trust;  

decision  making  is  done  jointly  

33

!  Cohesiveness of relationships increased from .62 to .71 on a scale of 0 to 1, almost a15% increase ◦  And this occurred in just 6 months

!  Most of the increase due to increased number of networking relationships

34

Organization/ Agency

Activity

Provided space Provided outreach and/or materials

Made referrals to agency

Co-hosted an event

Provider 1 " " " " Provider 2 " " " " Provider 3 " " " " Provider 4 " " " "

!  Name agency and check off type of activity !  Analysis: ◦  Count number of each type of relationships

35

!  Open-ended survey or interview ◦  “What four organizations has your program

collaborated with the most? Briefly describe the nature of each collaboration.”

◦  “What is the biggest lesson you have learned from

your collaboration?”

36

“No single agency can accomplish everything that providers do together”

•  Enhanced Client Service Delivery •  Better and more comprehensive services •  Bridge the strengths of each resource •  Collaboration is key to healthy community •  Increased referrals

! 

•  Positive Outcomes for Coordinated Efforts

•  Regular planning meetings beneficial •  Frequent contact and working together •  More can be achieved when no one

agency seeks credit

37

What has been your experience with collaboration in your county? To what extent are providers collaborating? Systems collaborating?

38

A.  Assessing prevention B.  Program variability C.  Funder and provider capacity D.  Creating tools E.  Examples F.  Keep in mind

39

!  Challenge to document something DIDN’T happen

Protective

Factors

Risk Factors

40

Type • Media

Campaigns • Mental Health

Promotion • Early

Intervention

Structure • Ongoing/

multi-session • Single

session/workshop

• Case management

Population • Adults • Youth • Children • Trainees

41

• Data collection • Evaluation/

research infrastructure

Funder

• Data collection • Data entry • Data analysis • Time

Provider

42

!  Assessment ◦  Program outcomes ◦  County/contractual requirements ◦  MHSA goals ◦  PEI objectives

!  Unit of analysis ◦  Individual programs ◦  Initiative ◦  System

!  Choosing indicators ◦  Usefulness ◦  Need to know

43

!  Focus: only Prevention programs !  Tools employed ◦  Post-only surveys !  Surveys varied by type of service (seminar/workshop, group,

case management) !  Surveys varied by type of program (e.g., substance abuse,

juvenile justice) ◦  Year end report (all Open-ended items)

!  Data collection ◦  Evalcorp trained on administration ◦  Evalcorp did data entry (thousands of svys)

!  Reporting findings ◦  Collaborative effort with Department

44

!  Prevention and early intervention programs !  Tools employed ◦  Pre-post standardized instruments for early

intervention (provider chooses) ◦  Pre-post for selective programs !  Adult, Adolescent, Child Versions ◦  Post-only for universal programs !  Adult, Adolescent, Child Versions

!  Data entry ◦  Evalcorp trained on administration

and data entry ◦  Providers conducted data entry

45

!  Data collection activities already underway !  Diverse tools being used !  Three initiatives ◦  Stigma and Discrimination Reduction ◦  Suicide Prevention ◦  School-base/Parenting

46

!  Early interventions ◦  Standardized tools (Ohio Scales of child symptoms

and functioning; PHQ-9) !  Existing program evaluation/workshop

evaluation tools !  Newly developed post-only outcome

measures !  Website analytics !  Open-ended survey about successes, challenges, overcoming challenges

47

• # trained to intervene in crises • Improved knowledge and

confidence in ability to intervene

More residents equipped to intervene

to prevent suicide

• # trained to intervene in crises Improve identification

of suicidal ideation among individuals receiving services

• Decreased suicide attempts among those in treatment

Reduce suicide attempts and

associated behavior

Objective Indicator

48

Goal: Reduce suicide rate

!  Rapport and cooperation with providers is essential

!  Be flexible and look for commonalities across instruments being used

!  Make strategic decisions about what and how to measure

!  If providers do data entry, ensure capacity, train carefully, check progress, maintain communication

!  Effective evaluation: participatory/collaborative from start to finish

!  Begin with the end in mind

49

50

Comments and Questions

Kristen Donovan, PhD. kdonovan@evalcorp.com

Joan Twohey-Jacobs, Ph.D.jtwoheyjacobs@evalcorp.com

51

www.evalcorp.com