Are We Making a Difference Feb 2013.ppt › pdf › making_a_difference.pdf · 2/11/2013 Brenda Herchmer [email protected] ©Grassroots Enterprises 2 THINKING ABOUT OUTCOMES Outcomes

2/11/2013

Brenda [email protected]©Grassroots Enterprises 1

2013 YUKON SPORT AND RECREATION LEADERSHIP SUMMIT

BRENDA HERCHMER

ARE WE MAKING ADIFFERENCE? DIFFERENCE?

…measuring and evaluating the outcomes of our

programs and services

SESSION OUTCOMES

Upon completing this webinar participants will be better able to:

use outcome measurement as a vehicle for ensuring we are making a difference

communicate the value of outcome measurement

describe what can be quantified and measured

identify strategies, models, and tools for measurement

TO BEGIN…

it isn’t about the actual measurement, it is ensuring we have something worth measuring

its not about more emphasis on measurementmeasurement

it is about stepping outside our comfort zones and thinking about what is possible

we will all have to change the way we see the work we do from “what we do” to “why we do it”

2/11/2013


THINKING ABOUT OUTCOMES

Outcomes are about: determining whether or not you

are making a difference creating a better futurec ea g a e e e the conditions you want to create

for: individuals served by your

organization (micro) your community as a whole (macro)

"In God we trust, all others bring data."- W. Edwards Deming

"Measurement is the first step that leads to control and eventually to improvement. If you can't measure something, you can't understand it. If you can't understand it, you can't control it. If you can't control it, you can't improve it."- H. James Harrington

"Without a standard there is no logical basis for making a decision or taking action."-Joseph M. Juran

"You get what you measure. Measure the wrong thing and you get the wrong behaviors."- John H. Lingle

THE REALITY OF TODAY’S WORK

ENVIRONMENTS

increased accountability new and often complex

challenges reduced resources pressures to enhance

efficiency and effectiveness

2/11/2013


THE CHALLENGE OF MEASURING

objectives may be vague impact is hard to measure accountable to many stakeholders management structures are intricate volunteerism is an important ingredient ROI is not always measured by dollars driven by value

“Nowadays people know the price of everything and the value of nothing”

Oscar Wilde

THIS HAS RESULTED

IN DEMANDS FOR...

strategic or business plans standards (Service Excellence) outcome or benefits-based

management systems new governance models new governance models performance

auditing/benchmarking evidence of return on

investment (ROI’s) emphasis on capacity building demonstration of contribution

to “social capital”

BUT ULTIMATELY…

what gets measured gets done if you don’t measure results,

you can’t tell success from failure

if you can’t see success you if you can t see success, you can’t reward it (and you end up rewarding failure)

if you can demonstrate results, you can win public support

Osborne & Gaebler, Reinventing Government

2/11/2013


WHY FOCUS ON OUTCOMES?

in an era of accountability, it is more important than ever that we promote and measure the value/benefits/end results of programs and services delivered by the voluntary and government sectors and their relationship to our g pindividual, social, economic, and environmental well-being

WHY EMPHASIZE

OUTCOME MEASUREMENT?

THE VALUE OF OUTCOME

MEASUREMENT communicates powerful results to

stakeholders in annual reports, budgets, marketing materials, proposals etc.

clarifies intended purpose and direction motivates staff identifies successful practices identifies successful practices

clearly positions organization improves performance assists process of resource allocation strengthens accountability helps identify potential partners encourages innovation

identifies training needs

2/11/2013


AN EMPHASIS ON OUTCOMES

HELPS STAFF AND VOLUNTEERS…

better understand the role they play in the pursuit of broader community goals and priorities

keep focused on delivering outcomes support necessary change support necessary change

ACTIVITY #1

1. How do you currently communicate the value of your organization? When elected officials, funders, board members, staff, volunteers, or clients ask you what results your organization produces or what impact it has had on the individuals you serve, how do you respond? What information or data do you provide?y p

2. What are the most useful pieces of information you collect to help you increase the effectiveness of your services?

TYPICALLY...

use somewhat limited business criteria (# of participants, cost/subsidy per person)

use quantitative vs qualitative data typically measure outputs instead typically measure outputs instead

of outcomes e.g. number of registrants, kilometres of bike paths, revenue levels

need to generate additional and complementary outcome-oriented indicators and measurement tools

2/11/2013


"What gets measured gets done, what gets measured and fed back gets done well, what gets rewarded gets repeated."

- John E. Jones

DEFINING

“OUTCOME MEASUREMENT”

“The regular, systematic tracking of the extent to which stakeholders experience the benefits or changes intended” (the outcomes!)

DEFINING OUTCOMES

outcomes are about creating a better future outcomes are about the conditions you want to

create both for individuals in your community (micro), and for your community as a whole (macro)(macro)

desired end results why does your organization/department exist? what difference would it make if you weren’t here

tomorrow? SO WHAT??

2/11/2013


THE 5 WHY’S

a technique that helps you to get past the what and how-to’s to the outcomes

very often, the answer to the first "why" will prompt first why will prompt another "why" and the answer to the second "why" will prompt another and so on; hence the name the 5 Why’s strategy

OUTCOME-BASED ORGANIZATIONS

focused on WHY it does programs and services, not what or how it does them (leadership vs management, “ends” vs the “means”)

see the forest and the trees

values facilitation, partnerships, and helping build capacity , p p , p g p yin others as a way of achieving and sustaining changes (outcomes) in the community

committed to supporting the community’s efforts to contribute to its own development (community development)

all of the organization’s programs, services, and functions are relevant to the community’s needs and grounded in shared outcomes

21

2/11/2013


22

23

24

2/11/2013


25

26

2/11/2013


28

2/11/2013


MISSION STATEMENTS

ACE COMMUNITIES To enhance the quality of life in rural Alberta by

strengthening community leadership, collaboration, and innovation through recreation, parks, arts, culture and heritage.

POSITION DESCRIPTIONS

E-Learning Centre CoordinatorPosition DescriptionACE Communitieswww.acecommunities.ca (an initiative of the Alberta Recreation and Parks Association)

Position DescriptionThe online Learning Centre is a key component of an ACE Communities/ARPA training and sustainability strategy designed to strengthen and develop community and recreation leaders across the province of Alberta and beyond. It is an ideal position for someone who wants to make a difference and grow with a challenging initiative. While the Coordinator will initially focus on (1) coaching content experts / presenters, (2) hosting the virtual classroom / meeting room (Adobe Connect), and (3) administration, it is also anticipated the Coordinator will play a key role in contributing to, and potentially leading, (4) the long term strategic growth and sustainability of the Centre.

MEASUREMENT NEEDS TO HAPPEN AT

DIFFERENT LEVELS…

Macro (how are we impacting the broader community good? it begins with the vision for your community)

Micro (how are our programs or services Micro (how are our programs or services impacting individuals?)

2/11/2013


MACRO

What future do you want to create for your community?

What positive conditions do you want to see in your community?

Wh t f t ld lik t t f th What future would you like to create for the individuals you serve?

What positive conditions would you like to create in their lives?

What would success look like if you had a crystal ball and could look deep into the future - for the individuals you serve, and for the community as a whole?

http://www.help4nonprofits.com

EXAMPLE: MACRO OUTCOMES FOR

PARKS & RECREATION

http://benefitshub.ca/ 50 outcome statements 750+ research studies reflects the personal, social,

i d i l economic and environmental benefits of parks, recreation, arts, culture, heritage, sports, active living

2/11/2013


VISION

Our London is a City ofneighbourhoods.

Our London neighbourhoods will be

London Strengthening Neighbourhoods Strategy

Our London neighbourhoods will beempowered, sustainable, safe and

active communities. We will care for and celebrate each other while encouraging diversity and inclusiveness.

Our neighbourhoods will be environmentally and socially responsible

and will have available green space, vibrant local economies

and accessible amenities of daily life.

Short TermOutcomes

Intermediate Outcomes

Program Goal Community members have the best possible quality of life in a strong and vibrant neighbourhood.

Resident s and community stakeholders have increased awareness and knowledge about services & supports available in the

Residents and community stakeholders have increased awareness and knowledge of

Increased integration and mobilization of partnerships and resources

The impact of poverty is reduced

Improved literacy

Increased healing, resilience and recovery for those living with

mental health issues

Increased healthy eating and healthy physical

activity

Long Term Outcomes

All community members have pride, ownership and sense of

belonging

All community members are empowered to reach their full

potential

All community members have knowledge and access to information

they need

Resident s and community stakeholders have increased

Link to community /

funder priorities

Resident level Increased empowerment of all community members

Increased engagement of all community

members

Reduced isolation

Reduced barriers to inclusion

Information & Referral

System level

Comm

unity D

evelopment

Activities

about services & supports available in the neighbourhood and community

awareness and knowledge of services & programs at NRCs

Core Services

Information & Referral

Community Building

Population specific

programs

Basic Needs

Increased access to basic needs

Quality activities are inclusive and responsive to the needs of the residents and the neighbourhood

connection to services and community activities

Increased skill development (i.e., parenting, literacy, language, employment, healthy lifestyle)

Increased personal development and leadership (i.e., self image, decision making, life skills, self advocacy)

Increased collaboration and networking at the neighbourhood level

Community Building

Pop’n specific programs

Basic Needs

It is recognized that, while program delivery approach

will vary between neighbourhood resource centres (NRCs), there are common core activities

Advocacy & Support

Advocacy & Support Residents feel supported

tApproach

MICRO the outcomes being delivered by programs, services,

events, and activities also need to be measured the difference your program will make in someone’s life how will life be better? for whom? the outcomes your stakeholders consider to be most

important helps determine where you fit within the pursuit of

broader community goals and priorities may be different from what you are currently delivering needs to be done so programs, services, events, facilities

etc. can be prioritized within the context of broader public good

2/11/2013


OUTCOMES DELIVERED BY PROGRAMS, SERVICES, EVENTS – MICRO LEVEL

what benefits or changes are there for stakeholders during or after their involvement with a program, service, or event?

what didn’t they have when they started? typically there are benefits or changes in:

1. knowledge2. attitudes3. values4. skills5. behaviour6. condition7. status

a change for the better (or in some cases - get worse more slowly than they would have otherwise)

Benefits or

OUTCOME MODEL(AKA LOGIC MODEL OR RESULTS CHAIN) SOURCE: UNITED WAY OF AMERICA

INPUTS OUTPUTS OUTCOMES

Resources What is done The direct

ACTIVITIES

changes for stakeholders e.g. new knowledge, increased skills, changed attitudes or values, modified behaviour, improved conditions, altered status

dedicated to or consumed by the program or service e.g. money, staff, volunteers, facilities, equipment,

supplies.

with the inputs to deliver your purpose/ mission e.g. marketing, registration, program admin, volunteer recruitment

products of the program or service activities e.g. number of classes taught, number of participants, sessions, volume of materials distributed

KEY CHARACTERISTICS

OF OUTCOMES

state only a single end state or result specifies the expected direction and amount of

change performance based (what change in whom by

when)when) if you control it or do it, then it is an output or an

activity if you can only influence it, it is an outcome

2/11/2013


SAMPLE OUTCOMES

1. “know the value of sport” (knowledge)2. “recognize that an appreciation of active living is

necessary for future success” (attitude)3. “recognize the importance of leisure literacy” (value)

“ bl t i l t f d i i l ” ( kill)4. “are able to implement a fundraising plan” (skill)5. “use verbal rather than physical means to resolve

conflict” (behaviour)6. “have an increased sense of community” (condition)7. “participate independently in recreation settings”

(status)

ACTIVITY # 2Which of the following are considered outcomes? (check all that apply)

Volunteer Management Program

representatives from 20 organizations attend the workshop six group workshops are conducted participants understanding of recruiting volunteers increases participants provide appropriate volunteer management participants participate in role plays and group discussions

Children’s Recreation Program

children master new individual and group activities 15 at-risk children attend the program activities are designed to encourage cooperative play children’s social skills improve children make more positive use of free time outside of the program

Special Event

200 people attend revenue is increased by 10% activities encourage family fun our sense of community is strengthened more people are aware of the value of our organization

CATEGORIES CAN BE FURTHER

SUB-DIVIDED OR SORTED ACCORDINGTO INCREASING LEVELS OF COMPLEXITY:

BLOOM’S TAXONOMY(lowest to highest)

RECALL KnowledgeC h iComprehension

UNDERSTANDING ApplicationAnalysis

PROBLEM SOLVING SynthesisEvaluation

2/11/2013


Learning Level –RECALL OF INFORMATION

ASSOCIATED ACTION WORDS

Knowledge define, describe, state, list, name, write, recall, recognize, label, underline, select, reproduce, outline, match

Comprehension identify, justify, select, indicate, illustrate, represent, name, formulate, explain, judge, contrast, classify

46

Learning Level –UNDERSTANDING


Application predict, select, assess, explain, choose, find, show, demonstrate, construct, compute, use, perform

Analysis analyze, identify, conclude, differentiate, select, separate, compare, contrast, justify, resolve, break down, criticize

47

Learning Level –PROBLEM SOLVING


Synthesis combine, restate, summarize, argue, discuss, organize, derive, select, relate, generalize concludegeneralize, conclude

Evaluation judge, evaluate, determine, recognize, support, defend, attack, criticize, identify, avoid, select, choose

48

2/11/2013


ONCE OUTCOMES ARE IDENTIFIED...

necessary to develop indicators for each outcome (how will you know the outcome is being delivered?)

develop measurement tools so progress towards outcomes and indicators can be quantified outcomes and indicators can be quantified

then reward and invest in programs, services, and behaviours that best deliver the outcomes

A GOOD INDICATOR WILL:

be directly related to the outcome (if the indicator is changing, the outcome is clearly being affected)

be quantifiable (e.g. begin with “number of”, “percent of”, “ratio of”, “incidence of”)

be specific in wording be specific in wording be positive whenever possible (motivational &

inspiring) relate to something we know we can change over

the course of the program, service or intervention (i.e. flexibility is easier to deliver than increased bone mass)

GOOD INDICATOR (CONT’D)

relate to something that is important to the customer, not just something that is easily measured

practical and worthwhile – add more value than they cost to collectthey cost to collect

reliable cost effective

Source: RETHINK (West) Inc.

2/11/2013


A GOOD MEASUREMENT TOOL WILL:

be credible (respected by researchers, other fields, etc.)

be quantitative – even if the change is qualitative, it is important to have a measure of the perceived difference (e g pre and post the perceived difference (e.g. pre and post application of perception scales)

minimize the influence that bias or ‘desire to please’ cannot change or influence the outcome in any significant way

be sensitive to small changes (e.g. a 7 point scale allows for finer calibration than a 5 point scale)

A GOOD MEASUREMENT TOOL (CONT’D)

allow for additional comment or clarification from the customer from the customer and/or staff (if only to identify additional dimensions or outside influences)

will be manageable – the test cannot demand too gmuch of either the customer or the staff involved

consider several perspectives – triangulation, for example, might get feedback on changed behavior from the child participant, from the instructor, from the parent, and from a teacher.

Source: RETHINK (West) Inc.

ACTIVITY #3DEVELOPING OUTCOMES FOR PROGRAMS, SERVICES, EVENTS…

1. clarify your stakeholders (clients, customers, participants, member, service recipients, intended beneficiaries).

2. select a program, service, or event you provide. 3 ask what benefits or changes for stakeholders 3. ask what benefits or changes for stakeholders

you want to deliver. What is it you want to be able to deliver in terms of:

new knowledge increased skills changed attitudes or values modified behaviour improved conditions altered status

2/11/2013


BEST APPROACH

Start with the final outcomes and move to key activities and outputs

“Begin with the end in mind”

STRATEGY

Outcome Indicator Source/Method

Responsibility Timing/Frequency

56

EXAMPLE

Orientation for Niagara CollegeRecreation and Leisure Services Students

(a 2 day off-site camping trip)Outcomes:• increased sense of “community” and comfort and potential for student

d t h i t tiand teacher interaction• students who are able to reliably demonstrate the ability to:

explain the benefits of recreation state the vocational learning outcomes of the recreation and leisure services program standard design and implement strategies for working with small groups

2/11/2013


Outcome Indicator Source/Method Responsibility

Increased sense of “community” and comfort and potential for student and instructor interaction

Student can name all instructors

Students can recall the names of a minimum of five other first year students

Students can recall the names of five second year (mentor) students

Icebreaker Activities and Name Games (carried on throughout the 2 days)

Judy

Explain the benefits of recreation

Students can list the 8 benefits messages (outcome categories that recreation has the potential to deliver)

Benefits video, review of Benefits Catalogue

Shared “storytelling”

Brenda

the potential to deliver) Students can provide one

specific benefit for each of the 8 eight categories

Shared storytelling session of benefits participants have personally derived from sports, recreation and cultural activities

Matching game played in teams (to match 44 outcomes within 8 categories)

State the vocational learning outcomes of the recreation and leisure services program standard

Students can recall a minimum of 7 of the 11 vocation learning outcomes

Distribution of program standard manuals

Team quiz

Christine

Design and implement strategies for

Student can explain the components of a successful group

Teams of students rotate to participant in experiential

David

strategies for working with small groups

group Explain why a group is not

successful Explain the types of roles

participants can play within a group

Apply a problem solving model

in experiential learning activities

Extensive debriefing

2/11/2013


CHALLENGES IN IMPLEMENTING

OUTCOME MEASUREMENT

it is a paradigm shift it isn’t easy it takes time it requires resources it requires some expertise some outcomes are more difficult to measure

than others some outcomes take longer to measure some funders will ignore all of the above

But…it will be the best investment you ever make!

ADDITIONAL RESOURCES

VOIS Database from the UK – more about selecting outcomes and getting ideas of indicators

http://www.sroi-uk.org/vois-database

SROI Canada’s Databasehttp://www.calgary.ca/CSPS/CNS/Pages/FCSS/Social-Return-

on-Investment-(SROI).aspx

http://www.thesroinetwork.org/component/option,com_docman/task,cat_view/gid,23/Itemid,38/

http://www.ssireview.org/articles/entry/measuring_social_value

QUESTIONS?

63

2/11/2013


for your time and energy today!BRENDA HERCHMERemail: [email protected]: brenda.herchmer.nettwitter: @brenda_herchmerphone: 780.488.8136facebook.com/brenda.herchmer

64

Documents

Are We Making a Difference Feb 2013.ppt › pdf › making_a_difference.pdf · 2/11/2013 Brenda Herchmer [email protected] ©Grassroots Enterprises 2 THINKING ABOUT OUTCOMES Outcomes