Writing & Measuring Consumer Education & Referral Outcomes

1

A QTS Web Training

Writing & Measuring Consumer

Education & Referral Outcomes

2

Things to know

You should have the Power Point open on a computer in front of you

You have control of the power point on your own computer

The trainer will tell you when to go to the next slide

3

Conference Call Etiquette….

Please mute your phone unless you have a question or comment

Please do NOT put your phone on hold during the call

If we should get disconnected by phone, call back in

4

Details . . .

Attendance will be taken at the conclusion of the call

If you need additional assistance, please work with either your:– Quality Assurance Training & Support Specialist, or– Regional Support Services Manager

5

What we will do…

Review expectations of 2.B.1.c and 2.C.1.a

Review outcome measurement basics

Discuss realities of outcome measurement

Anything else?

6

Results Accountability Guidebook

Results Accountability Guidebook Child Care Resource Referral, Oregon State University Family Policy Program & Oregon Child Care Research Partnership, Pratt, Clara, and Weber Bobbie, 2000.

Chapters 2, 3, and 6 http://www.naccrra.org/docs/results_accountability_guidebook_10-00.pdf. 

7

2.B.1.c

An outcome based plan drives your services Appropriate outcomes, indicators, activities,

and inputs Evaluation tool used to measure indicators Compilation of that data (Based on what you

are collecting, how do you know you’ve achieved the outcome?)

8

2.C.1.a Outcome Based Evaluation

Tools measure progress toward outcomes

Surveying consumers (minimum 20% response rate)

Compiles and analyzes data

Uses survey results to inform services

9

Principles of Performance Measurement

Know what performance measurement is. Begin with a program logic model. Balance utility, feasibility, accuracy, and respect. Identify the capacity for performance measurement. Choose a design. Use what is learned.

Results Accountability Guidebook, pg. 81

10

Performance Measurement includes answering the following ?s:

What are the planned activities, outputs, and intended outcomes?

Does the program achieve its planned activities and outputs?

Does the program achieve its intended outcomes and results?

Results Accountability Guidebook, pg. 82

11

Logic Model Results Accountability Guidebook, pgs. 18

12

Logic Model

1. Identify a goal2. Identify one initial and one long term outcome3. Identify the key program activities that will lead to the

achievement of those outcomes4. Specify the volume and quality of outputs5. Identify the major resources or inputs needed6. Realism check7. Identify indicators of measures

Results Accountability Guidebook, pgs. 18 & 19

13

Think of it in this way

Outcome—what you want to happen; specific, observable, measurable, and achievable

Activity—actions or strategies that are going to help you reach your outcome

Indicator—measures qualitative change (outcomes)

Outputs—measure activities in terms of volume or quality

14

What’s the difference?

Outcome: Increased knowledge of child care options that meet family needs

Activity: Use CCR&R database to link families with providers appropriate to family needs

Outcome Indicator: 75% of parents responding to the follow up survey report an increase in knowledge of child care options that meet their needs (survey)

Output Indicator: 256 families received referrals in the month of May. 97% of clients received between 5 and 10 child care options. (database)

15

What is an outcome?

Answer this question:

What is it you want to happen to clients who call your referral line?

Think about how you measure if it has happened.

16

Outcomes are the program’s results for people & communities

Contribute to the achievement of longer term goals

Describe a desired state of knowledge, attitude, skills, behaviors, or circumstances

Outcomes are observable, measurable, and achievable

Results Accountability Guidebook, pg. 24

17

Outcomes seek to…

Increase

Decrease

Maintain

Results Accountability Guidebook, pg. 24

18

Sample Outcomes

InitialIncreased knowledge of the characteristics of high quality child care

Longer termIncreased family satisfaction with child care

Results Accountability Guidebook, pg. 24

19

What is the difference?

Outcome

Output

20

What’s the difference?

Outcome

Increased knowledge of the characteristics of high quality child care

Output

55 parents received quality indicator brochures in the month of December.

21

The difference

Outcome—states the desired changeIncreased knowledge of the characteristics of high quality child care

Output—volume and/or quality measure of what occurs as a result of activities. Volume: Number of families who received written materials describing characteristics of high quality child careQuality: Parent satisfaction with consultations and educational materials about quality child care characteristics

22

Another example

Outcome

Increased knowledge of child care options that meet family needs

Output

Volume: Number of families who received child care referrals

Quality: Accuracy of referrals received by families

23

Performance Measurement

Compares planned activities, outputs, and outcomes to: – Actual accomplishments– Accomplishments of other programs– Population norms

Uses simple design for data collection Includes all or a good sample of participants

24

Performance Measurement (continued)

Uses records, observations, surveys

Relies on descriptive data analysis

Informs service delivery and demonstrates results

Results Accountability Guidebook, pg. 83

25

Decide what you will measure

“Not everything that is measurable is important; not everything that is important is measurable.”

Michael Patton, Evaluator

Results Accountability Guidebook, pg. 88

26

How do you decide what to measure?

What info is most needed to improve program operations and outcomes?

What will produce the most accurate information? What is feasible given staff and participant time, staff

skills, and program resources? What is worthwhile and respectful of participants and

staff?Results Accountability Guidebook, pg. 89

27

Identify your capacity

Level A

Level B

Level C

Results Accountability Guidebook, pg. 90

28

In deciding what to measure, the so that chain can help

Key Activity: Use CCR&R database to link families with providers appropriate to family needs, so that

Outcome 1: Families have increased knowledge of child care options that meet their needs, so that

Outcome 2: Families have increased access to child care options that meet their needs, so that

Outcome 3: Families use information to find child care, so thatOutcome 4: Families are satisfied with child care arrangement, so thatOutcome 5: There is an increased probability of continuity of care for the

child, so thatUltimate goal: Increased family sufficiency and ability to achieve personal

goals is achieved.

29

What is it you notice as you go further up the chain?

Key Activity: Use CCR&R database to link families with providers appropriate to family needs, so that

Outcome 1: Families have increased knowledge of child care options that meet their needs, so that

Outcome 2: Families have increased access to child care options that meet their needs, so that

Outcome 3: Families use information to find child care, so that

Outcome 4: Families are satisfied with child care arrangement, so that

Outcome 5: There is an increased probability of continuity of care for the child, so that

Ultimate goal: Increased family sufficiency and ability to achieve personal goals is achieved.

Key Activity: Provide information about the characteristics of high quality child care, so that

Outcome 1: Parent knowledge of characteristics of high quality child care increases, so that

Outcome 2: Parents use knowledge of characteristics of high quality child care in their child care search, so that

Outcome 3: Parents choose high quality child care arrangement, so that

Outcome 4: Parents are satisfied with child care arrangement, so that

Outcome 5: There is an increased probability of continuity of care for the child, so that

Ultimate goal: Families experience increased family sufficiency and ability to achieve personal goals is achieved.

30

What is feasible to measure?

Key Activity: Use CCR&R database to link families with providers appropriate to family needs, so that

Outcome 1: Families have increased knowledge of child care options that meet their needs, so that

Outcome 2: Families have increased access to child care options that meet their needs, so that

Outcome 3: Families use information to find child care, so thatOutcome 4: Families are satisfied with child care arrangement, so thatOutcome 5: There is an increased probability of continuity of care for the

child, so thatUltimate goal: Increased family sufficiency and ability to achieve personal goals is

achieved.

31

And these?

Key Activity: Provide information about the characteristics of high quality child care, so that

Outcome 1: Parent knowledge of characteristics of high quality child care increases, so that

Outcome 2: Parents use knowledge of characteristics of high quality child care in their child care search, so that

Outcome 3: Parents choose high quality child care arrangement, so thatOutcome 4: Parents are satisfied with child care arrangement, so thatOutcome 5: There is an increased probability of continuity of care for the

child, so thatUltimate goal: Increased family sufficiency and ability to achieve personal

goals is achieved.

32

At the very least…

…agencies should track their initial outcomes of key activities.

33

Choose a design

When will information be collected? From whom will information be collected? What information will be collected and how will

it be analyzed? How will participants’ needs, rights, and

welfare be protected?Results Accountability Guidebook, pg. 92

34

Most common

Database

Surveys– Written– Telephone

35

How does your survey help measure your outcome?

Key Activity: Use CCR&R database to link families with providers appropriate to family needs, so that

Outcome 1: Families have increased knowledge of child care options that meet their needs, so that

Outcome 2: Families have increased access to child care options that meet their needs, so that

Outcome 3: Families use information to find child care, so that

Key Activity: Provide information about the characteristics of high quality child care, so that

Outcome 1: Parent knowledge of characteristics of high quality child care increases, so that

Outcome 2: Parents use knowledge of characteristics of high quality child care in their child care search, so that

Outcome 3: Parents choose high quality child care arrangement, so that

36

Were our services helpful?

Outcome 1: Families have increased knowledge of child care options that meet their needs, so that

Outcome 2: Families have increased access to child care options that meet their needs, so that

Outcome 3: Families use information to find child care, so that

Outcome 1: Parent knowledge of characteristics of high quality child care increases, so that

Outcome 2: Parents use knowledge of characteristics of high quality child care in their child care search, so that

Outcome 3: Parents choose high quality child care arrangement, so that

37

Did you find child care?

Outcome 1: Families have increased knowledge of child care options that meet their needs, so that

Outcome 2: Families have increased access to child care options that meet their needs, so that

Outcome 3: Families use information to find child care, so that

Outcome 1: Parent knowledge of characteristics of high quality child care increases, so that

Outcome 2: Parents use knowledge of characteristics of high quality child care in their child care search, so that

Outcome 3: Parents choose high quality child care arrangement, so that

38

An example

Outcome 2: Families have increased access to child care options that meet their needs

Indicator: 85% of parents report that CCR&R consultation helped them to find appropriate child care arrangements, even if the setting found was not one of CCR&R referrals

Sample Survey Question: Did you eventually use one of the providers referred to you by the CCR&R for this

child? Yes NoIf yes, are you still using this provider? Yes No How long?If no, What did you eventually do? Give an array of choice

Did your contact with the CCR&R help you make a more informed choice when selecting child care for your child? Yes No

39

Another example

Outcome 1: Parent knowledge of characteristics of high quality child care increases

Indicator: 85% of parents responding to the survey report an increase in their knowledge of the characteristics of high quality child care as a result of their contact with the CCR&R

Sample question:How would you rate your current understanding of things to consider

in choosing good quality child care? 1 2 3 4 5 N/AHow would you rate your understanding of these things before you

talked with the CCR&R? 1 2 3 4 5 N/A

40

How could you measure these?

Increased usage of referral services by families seeking child care and increased public awareness of the CCR&R services

Increased usage of quality indicators when choosing child care

41

Key points

Start with a logic model– What do you want to accomplish? (outcome)– How will you know if it has happened? (indicator)

Proceed realistically Choose a design Use what is learned

42

Implement these strategies to avoid common errors…

Understand terminology– Areas of particular confusion outcomes vs. activities;

outcomes vs. outputs; indicators vs. outputs Include only outcomes that are realistically measurable

and that you are or will be measuring Make sure the tool that you are using is actually

measuring what you say it is Include a compilation of data that supports

achievement of the outcome

43

What we wanted to do

Review expectations of 2.B.1.c and 2.C.1.a

Review outcome measurement basics

Discuss realities of outcome measurement

44

Wrapping up

Attendance– Please STAY ON the line until everyone’s name is

called, otherwise we hear only beeps!– If your name is NOT called and/or you are sharing

the connection with someone else, please email me at Gail.Upton@naccrra.org with:

Your name, CCR&R name, position title, e-mail address, and Agency ID

45

Contact Info

Questions:

Gail.Upton@NACCRRA.org

OR

(866-782-2261)