From Outcomes to Impact How to Create Change in Family Violence

From Outcomes to ImpactHow to Create Change in Family Violence

Dr. Robbie Babins-WagnerCEO, Calgary Counselling Centre

Adjunct Professor, Faculty of Social Work, University of Calgary

FCSAA Power of Prevention Conference, November 29. 2013

• For at least the past 10 years funders of non-profit organizations have required outcome data as a measure of accountability.

• Agencies are increasingly reporting on program outcomes on an annual basis.


• The outcome agenda has brought greater accountability to the sector but there exists a gap in moving from outcomes to impact

• How do we improve the overall outcomes for the clients that we serve.


My interest outcome and evaluation began almost 30 years ago when I worked in health care and extended to my work at Calgary Counselling Centre. It lead to my completing my doctorate in outcomes and now is a prime focus of my work at CCC and the collaborative work I am doing with colleagues through the Center for Clinical Excellence in Chicago


• It is also a focus of my work through the United Way of Calgary’s Leading Boldly Network

• The Network is focused on – Building new capacity in Calgary for Collaborative

Social Innovation (CSI), and– Making progress on complex problems that are

beyond the mandate of any individual organization.


• Research suggests that most agencies have some outcome data

• But even with having the data, we have not been successful in achieving significantly better outcomes at a magnitude that matches the need in critical areas such as mental health, child outcomes, school completion and more.


• Its suggested that – we have not marshalled the full extent of available

knowledge and applied it to complex problems…– which would in turn generate new knowledge to

address some of our toughest social problems.


Frustrations1. On the part of non-profit leaders who have

not been able to find support (financial and non-financial) from funders for collecting and using information to manage to outcomes.

2. On the part of funders who have become frustrated with non-profits who have no means of determining whether they re doing what they say they do.


• The research is clear, it take a lot on the part of both agency leaders and funders to support the changes being required to solve complex problems.

• Having a strong outcome platform is the first step for agencies.


• The focus on having a strong outcome platform requires a difficult cultural transition from simply having a genuine interest in improvement to truly infusing outcome thinking into the way we manage our organizations


• Some funders suggest that to do this properly, requires a full 10% of their available funds for direct investment (on their parts) while at the same time supporting agencies to alter their culture and develop the human and IT systems necessary to manage to outcomes (Morino, 2010).

• Where this was initially tested, only 5 of 12 (42%) agencies achieved this kind of transformative change


• There are funders (in the US) who are investing in non-profit leaders who embrace the value of great information, even it they haven't had the external support to build the systems they need for collecting and using information .


• Funders are increasingly focused on outcomes that lead to larger impact:

• They are:– asking grant applicants to demonstrate a predisposition

for using information to guide operations– providing longer term funding, with clear expectations

that it will take at least 2-3 years of intensive work to create a true outcome focused culture

– learning to be more flexible in how to support this work


• Funders have also learned: – They cant impose this type of change– They need to give the agencies the time and space

to do it their own way, if agencies choose to do it at all

– The results aren’t going to sustain if the agencies don’t own the process and resulting systems

– This work isn’t for everyone.


• And when non-profit agencies elect to do this work they: – do a better job of meeting their missions– become more focused and disciplined in managing

their organizations – Are enabled towards a path toward managing to

outcomes– strengthen internal culture of the agency


Moving from outcomes to impact will bring forth the following results: • staff will develop better skills in their core

capacity (i.e. counselling)• program managers will develop better people

management skills and have the tools they need• board members will have the information to

provide more effective oversightAll of which lead to better results for clients


Steps to get the process started: 1. Determine who you will get feedback from clients

(service users) 2. Learning must be the primary goal of the phase one

where you – Collect information about the problem– Collect information about possible solutions

3. Apply rigor within reason 4. Be practical5. Create a learning culture


• The ProblemDespite all the right intentions many non-profits do not have the benefit of; – good information– tools to determine where they are headed– mapping a logical course towards an objective,

and– course-correct when required.


– While these are not new problems, current economic conditions locally, provincially federally and globally suggest that we are likely to see an increased focus on results to support decreased funds for social and human services

– The research suggests that organizations which demonstrate meaningful, lasting impact will be the focus of increased investment over the next decade.


• There is also the suggestion that individual organizations who can demonstrate impact will be brought together with other like minded organizations to work together in disciplined ways toward collective impact (Kania & Kramer, 2011).

• An additional focus will be a focus on working across silo’s - within organizations, across organizations in the community and within government


As non-profits focus on improved client benefit, we need to:

– clarify the results (outcomes) we are trying to achieve– collect the information that can best help us navigate

towards these outcomes – differentiate between operational performance

(overhead costs) and organizational effectiveness (results or outcomes)

– ensure we have the culture to support our work


In Sum• Many non-profits have no reliable way to know

whether they re on track to deliver on their promise • We need to invest in continuous collection and use

of information (data) to guide program/agency decisions and operations

• This change requires a significant culture shift within an organization – it is primarily about culture and people (not numbers)


In Sum (cont.)• We must focus on why measure and on what

to measure – not just on how to measure• The non-profit needs to drive the change and

be the primary beneficiary of it• Reasonableness and common sense must

guide both agency and funder investment decisions


Tris Lumley, head of strategy for New Philanthropy Capital (London) suggests that

“Great organizations …. are build around great data. Data that [allow] then to understand the needs they address,

what activities are likely to best address these needs, what actually happens as a result of these activities, and how to

allocate resources and tweak what they do for even greater impact. Too often, funders set the agenda with their own

requirements [and]cripple the organizations they are trying to help.


So how do we do this? How do we manage to outcomes?


5 Steps• Develop clarity through reflection and analysis on what change

you are trying to create• Gain specificity on how you will accomplish this change• Determine what information (hard and soft) will be most helpful

for gauging whether you are on or off track to achieve that change• Collect and use this information to plan, make important

decisions, track, course-correct and improve• Combine the steps above with good practice judgement and keen

discernment, which are more important than any single metric .


Of most importance• Gain clarity on what change you are trying to achieve• Develop specificity on how you will accomplish this change• Determine what information will be most helpful for gauging

whether you are on or off course• Collect and use this information as the basis for

understanding what’s working, planning, decision making and improvement

• Lastly, the technology behind the systems is imporntat and necessary but is not nearly as important as the mindset of the leaders who put these systems in place


Organizational Culture• Recruit culture leaders • Walk the talk • Know what you stand for (core beliefs & guiding

principles)• Be able to answer the question “To what end”

– improved client results → community impact• Ensure that everyone is moving in the same direction• Be clear and direct about expectations• Encourage self-improvement and personal growth


A Case Example


The OQ 45• a 45-item self-report measure• takes about five minutes to complete• grade 6 reading level• provides a total score, based on all 45 items, as well

as three subscales: symptoms of distress, interpersonal relationships and social role functioning

• scores range from 0 – 180, with a clinical cut-off of 63


59 60 61 62 63 64 65 66 67 68 69 700

20

40

60

80

100

120

140

160

Oq 45.2 Total Score

Score

Cutoff Score

Session Number

Scor

e


Clinical Significance• Criteria for clinically significant change (Jacobson & Truax,

1991) are used to make judgements of client outcomes• Clients who change in a positive or negative direction by at

least 14 points are regarded as having made “reliable change”• The clinical cut-off of 63 differentiates clients in the

dysfunctional range from those of a functional population• Very high clinical standard to meet (very unlikely to happen by

chance)• Most clinicians are satisfied with a smaller point change in

outcome suggesting positive or negative movement.


Four Categories of Change for the OQCategory Definition

No Change Any score that sows neither reliable (14 point) improvement or deterioration.

Deteriorated The client’s score is at least 4 points worse than the first OQ ( reliable negative change)

Improved The client’s final OQ score is at least 14 points less than their first session OQ Score

Recovered The client’s final OQ score is at least 14 points less than their first session score and the client’s final OQ score is at or below the clinical cut-off of 63.


N Deteriorated %No

Change % Improved % Recovered %Total Improved & Recovered

Baseline 6072 8.2 56 20.9 14.1 35

CCC 2009

2052 147 7.2 1021 49.8 471 23.0 413 20.1 43.1

CCC 2010

2284 173 7.6 1085 47.5 519 22.7 509 22.3 45

CCC 2011

2469 160 6.5 1152 46.7 612 24.8 545 22.1 46.9

CCC 2012

2288 148 6.5 1023 44.7 597 26.1 520 22.7 48.8

Total 16680 1237 7.4 7909 47.4 3771 22.6 3390 20.3 42.9


2004 - 2012 Average OQ at First Session

Average OQ at Last Session

Average Change

Full Timen=2759

68.74S=25.13

59.42S= 26.61

-12.62S=20.76

Associaten=1528

69.73S=25.96

59.94S=27.45

-12.62S=20.63

ResidentN=5377

75.47S=25.69

64.99S=27.70

-13.25S=21.73

InternN=19153

75.95S=26.46

65.51S=28.66

-13.41S=21.46

TotalN=18,817

74.07S=26.14

63.81S=28.04

-13.32S=21.35


Why Family Violence • Everyone is vulnerable to family violence• It is a significant social problem (Johnson,

2006)• The financial cost to Canadian Society

– In 2009, the total economic impact of spousal violence was estimated at $7.4 Billion (Zhang, Hoddenbagh, McDonald& Scrim, 2012)


The 2013 Stats Can report (based on 2009 data).

• Only 22% of victims reported violence to the police• In Canada, in 2008, 334,000 individuals were victims

of self reported violence at the hands of an intimate partner – 6% of the adult population

• Alberta has the second highest rate of self reported abuse, followed by Saskatchewan.

• The consequences are significant for victims & children as well as the abuser


Demographic Predictors: • Younger • Living in a common-law relationship• Living in a step family

• Income levels and education levels were not predictors of family violence


CCC Male Family Violence Clients (2008 – 2012) On average • 35 years old, range of 18 – 76• Median income: $25,000-35,000• Marital status: Single (41.1%); Married/Common-Law

(38.9%); Separated/Divorced (19.9%)


Calgary Counselling Centre’s Responsible Choices for Men Program The primary goal is to assist men to become

violent free Major objectives include:

o decreasing all forms of abusive behaviour,o accepting responsibility for one’s behavior,o increasing self-esteem,o improving family relations,o decreasing stress,o increasing empathy towards the victim, ando stopping abuse towards children


Program Elements: • Counselling to engage in process

– Specialized 14 week (30 hour) Group – Counselling for any outstanding issues

• Outcome measures • OQ data is collected on a session by session basis during

counselling, at the beginning of group, at the end of group and at the end of counselling


Responsible Choice for Men Outcomes (2008-2012):

• 1545 clients (individual + group)• Of these clients, 607 went on to participate in a specialized

family violence group whereas 938 proceeded with counselling

• Average number of counselling sessions for those who participated in counselling and group = 6

– The range of counselling sessions for those who did not participate in group was 1-41.

– The range of counselling sessions for those who participated in group was 1-23.


RCM Counselling only Results2008 - 2012

Average First Session OQ

Average Last Session OQ

Change Score

Results

n=606 53.4 41.7 -11.7 t(602) = 13.31, p <.001*

Range of OQ scores

2-144 1-153


Combined ProgrammingRCM

N=299Unadjusted Score

N=292Adjusted score

Change ScoreUnadjusted

Change ScoreAdjusted

Results – Unadjusted scores

Results – Adjusted scores

First Counselling Session

45.2 85.7

First Session of Group

33.1 77.9 -11.78

Last Session of Group

28.6 71.4 -4.5 -6.5

From first session of counselling to last session of group

----------- --------- -16.6 -14.3 t(299) = 12.79, p <.001*

t(291)=10.6, p<0.001*


Combined Programming

N=299 Change Score

Results (Unadjusted)


45.2


33.1 -12.1


28.6 -4.5


-16.6 t(299) = 12.79, p <.001*


Combined Programming

N=292 Change Score

Results – adjusted for social desirability


85.7


77.9 -7.8


71.4 -6.5


-14.3 t(291) = 10.63 p <.001*


CategoryUnadjusted

Individual Counselling -

RCM

CCC Cases US Baseline Study

Deteriorated 43 (7%) 648 (7.6%) 8.2%

No Change 299 (50%) 4055 (47.4%)

56%

Improved 183(30%) 1933 (22.6%)

20.9%

Recovered 78(13%) 1915 (22.4%)

14.1%

Total Improved or Recovered

261(43) 3848(45%)

35%

Total 637 8,551 6,072


Category From first RCM Counselling session

to first group session


Deteriorated 31 (5.7%) 648 (7.6%) 8.2%

No Change 305 (56.1%) 4055 (47.4%) 56%

Improved 168 (30.9%) 1933 (22.6%) 20.9%

Recovered 40 (7.4%) 1915 (22.4%) 14.1%

Total Improved or Recovered

208 (38.8) 3848(45%)

35%

Total 544 8,551 6,072


CategoryUnadjusted

From first Family Violence

counselling session to last group

session


Deteriorated 11 (7.2.%) 648 (7.6%) 8.2%

No Change 62 (40.8%) 4055 (47.4%)

56%

Improved 58 (38.2%) 1933 (22.6%)

20.9%

Recovered 21 (13.8%) 1915 (22.4%)

14.1%

Improved or Recovered

79 ( 52%) 45% 35%

Total 152 8,551 6,072


CategoryAdjusted for social desirability

From first Family Violence

Counselling session to last group

session


Deteriorated 11 (7.2%) 648 (7.6%) 8.2%

No Change 62 (40.8%) 4055 (47.4%)

56%

Improved 58 (38.2%) 1933 (22.6%)

20.9%

Recovered 21 (13.8%) 1915 (22.4%)

14.1%

Improved or Recovered

79(52%) 45% 35%

Total 152 8,551 6,072


Combined Programming to counselling follow-up

N=152 (unadjusted)

Change Score

Results


45.3


33.1 -12.2

Last Session of Group 30.1 -3

Last Counselling Session

28.8 -16.5 T(151)=9.4, p<0.001*


Conclusion– Overall results support continued use of this

framework of individual and group– Decreasing the number of clients doing individual

counselling only – Will work with the program to support best results

for clients


[email protected]

www.calgarycounselling.com

Documents

From Outcomes to Impact How to Create Change in Family Violence