Evaluating Health Communication Strategies: Part 1tacenters.emory.edu/resources/products_tools/...A pet peeve of evaluators is, when someone has finished their program and they say,

Evaluating Health Communication Strategies: Part 1

Page 1 of 24

Evaluating Health Communication

Strategies: Part 1

Ms. McDivitt: We are at stage four. Now

you may remember that in the planning section

we had developed an evaluation plan. So, it is

not that we have forgotten the evaluation up

until this point.

A pet peeve of evaluators is, when someone

has finished their program and they say, “Come

and evaluate it. It ended last month.” We like

to be involved at the beginning. I will try to

show you how evaluators can provide value,

actually, in planning and implementing your

program, if you involve us at the beginning.

So, let’s go.

But I wanted to give you a definition of

evaluation research. This one is from the World

Health Organization. There are two things that

I wanted to focus on.

No. 1, systematic examination and

assessment of a program, the parts of a program,


Page 2 of 24

to produce information that can be used by those

who have an interest. I think a lot of times

people do evaluation and they don’t really think

about how it is going to be used, whether it is

going to be useful. We just sort of say, “Ah,

we are supposed to do an evaluation.”

I think if you don’t develop an evaluation

that is going to be useful to you and your

stakeholders, then you are missing a great

opportunity. To me, the whole usefulness is a

huge thing, useful in helping you make decisions

about what you want to do next, about whether

you want to continue this program at all, et

cetera.

Evaluation can help you do a lot of things.

I think we tend to think of evaluation as, did

the program work? But, in doing an evaluation,

you can monitor how well the program went, see

whether you met your objectives, document

strengths and weaknesses. You can use it to

refine your program, essentially, make

improvements.


Page 3 of 24

Another thing you can do, and that has been

done with a lot of evaluations, is adding to the

knowledge base. So that someone else can, then,

say, “Aha, they did this kind of program. They

had success. They didn’t have success. Here’s

what I might want to do myself.”

One of the ways that people talk about

evaluation is in terms of process evaluation,

outcome evaluation, impact evaluation. I

think in health communication, I mean, a lot of

the work that we do is really looking at how well

is the program functioning. Is the program or

the communication activities of high quality,

of acceptable quality, looking at costs. So,

essentially, a lot of the monitoring, the

documenting, as you go along.

Outcome is looking more at the kind of

intermediate behaviors, sometimes knowledge,

awareness, attitudes, and policy. Impact is

really the very long-term, looking at whether

your activities had an impact in terms of

health.


Page 4 of 24

I think, again, a lot of people think about

evaluation as just being outcome evaluation or

what is called summative evaluation. But I

would like to stress that it is important really

to do both process and summative. Because if

you just say, “Did my program work?” and you

don’t understand how, or if your program didn’t

work as you wanted it to and you can’t explain

why not, then you have missed an opportunity to

really learn.

We follow the CDC evaluation framework in

thinking about evaluation. There are a number

of steps. Again, as with the planning process,

they aren’t always exactly in order. Sometimes

you go back and forth.

But, essentially, starting by engaging

stakeholders, describing your program,

focusing your design. Again, in the planning

process we talked about how you really need to

try to focus. You can’t just fling messages out

everywhere. Gathering evidence, justifying

your conclusions, and making sure that your


Page 5 of 24

evaluation is used, and that you could share

lessons learned.

And again, the whole stress on use, making

sure that it does feedback. There is nothing

sadder than to see an evaluation report sitting

on a shelf covered with a layer of dust because

it wasn’t useful. It didn’t make a difference.

Another thing to think about is standards

for evaluation, particularly, again, standards

of usability, which tends to be one of my

favorites, but also feasibility. In thinking

about evaluation, is this a realistic

evaluation to do? Does it make sense? In terms

of the amount of money I have, does it make sense?

Legal and ethical issues. Anytime that

you are measuring things, talking to people, or

making judgments about a program, about what

people are doing, I mean evaluation is judgment.

We do evaluation every day. You look at what

is in your closet and you evaluate what am I going

to wear today. You make a judgment. You looked

at the courses that were offered here, and you


Page 6 of 24

at some level evaluated them. Now you didn’t

do a big study, but you did make a judgment as

to what was going to work for you. So, really

thinking about ethics and whether what you are

doing in your evaluation is really going to hurt

someone.

And then, accuracy, making sure that you

collect information that is accurate and

essentially worthy.

Okay. Step one is engaging stakeholders.

I think oftentimes this step is actually not

done a lot of the time. We don’t necessarily

think in a formal way about who is going to be

interested in this evaluation. Who is going to

use the results? Who is going to want to know

what we are going to find out in this evaluation?

So, who are some possible stakeholders in

your program, if you are evaluating your

program? Who might be interested in knowing the

results? A lot of times, if you don’t involve

the major stakeholders, when you finish your

evaluation and if they haven’t had any input,


Page 7 of 24

if you are not aware of what kinds of things they

need to know, you may not be measuring those

things. And so, you get to the end and you say,

“Here’s our evaluation,” and they say, “Well,

we don’t believe it. We think success is this,

and you say success is that, but the two are not

matching.”

Particularly with funders, if CDC is

saying, “We expect you to get 10 percent

change,” and, realistically, you really think

you can maybe get 2 percent, No. 1, the funders

need to be aware and perhaps have a little bit

of a reality adjustment. But, also, just coming

to some kind of agreement as to what success is,

and a lot of people are going to have different

opinions about what success is.

One thing to think about is, what kinds

of things would the program manager want to know

in an evaluation? That’s you. When we do an

evaluation, what do you want to know?

Participant: Is it working?

Ms. McDivitt: Is it working? So, it is


Page 8 of 24

not just, did it work? Is it working? So, that

is kind of a process question. You want to know

as you go along. You don’t want to know just

at the end because at the end you can’t fix it.

Okay?

What else do you want to know?

Participant: Do you need to make any

changes along the way?

Ms. McDivitt: Okay. So, are there

things that you need to do to improve it or is

it going off in the wrong direction? Are some

things not happening? Are you not having the

impact you wanted? So, again, this is more of

a process piece.

Legislators, do they want to know how it

is working?

Participant: Did it work?

Ms. McDivitt: Did it work? Okay. So,

they have got a very different interest. If you

start giving them information about, “Well, we

sent out this much of this,” and we did this and

we did that, they are going to be bored.


Page 9 of 24

However, they are going to want to know, did it

work?

Participant: And how it affected my

district.

Ms. McDivitt: And how it affected their

district. So, that is even more complicated.

I think one of the things is that you need

to think about what are all the different kinds

of stakeholders who need to be at the table who

are potentially going to use the results of your

evaluation, and what kinds of things do they

want to know. Now you can’t always answer

everything they want to know. But, at least if

you are aware, then you can think about it.

I think somebody said you want to involve

them throughout. You want to involve them at

the beginning. You want to do kind of reality

checks throughout. Of course, you want them to

be involved at the end.

I think something that is important here

is to identify at what stages do people want to

be informed. So, your program manager is going


Page 10 of 24

to want to be informed about more of the details

more often than a legislator or perhaps a

funder, or whatever. Again, trying to

understand some of the expectations and address

them.

Describe the program. This is actually

one of my favorite steps. This is where you

really want to understand, what is the program?

How is it working? What are the assumptions you

are making about how it is working? What is the

need for the program? What are the expected

effects? What do you think is going to happen

as a result of your activities? What are the

activities? What kinds of resources are you

putting into it?

Where is your program? Are you starting

a completely new program that has never been

done in your agency or are you using something

that is sort of a standard of practice? Or are

you sort of doing a tweak on a program you have

already done? So, kind of where you are in the

stage of development is going to matter.


Page 11 of 24

What is the context in which this is taking

place? What is the political context? What is

sort of the funding context of your agency?

Really kind of looking at the bigger picture.

A lot of this is what goes into a logic

model. I think a number of people think, “Oh,

yuck, logic model.” But, to me, this is how you

really can understand all the different pieces.

Because an evaluation is not just looking at the

very end. If we really want to understand how

the program worked, and if it worked, was it due

to what you did, and how did the whole thing come

about, then you need to think about some of these

other pieces.

So, really, trying to understand how you

are going to get there and, essentially, the

logic of the program, how you think things are

going to happen. Essentially, the logic model

kind of puts all the pieces together.

I would like to suggest that you actually

build your logic model in terms of what you think

is going to happen, whether you can measure it


Page 12 of 24

or not. Because you can note, okay, I can’t

measure this, and then you may have to think

about, okay, what are you going to do about that?

But I think it is very important to really think

about the whole process of how this is going to

work.

People have in their minds we are going

to do all these things, and it is going to result

in this. It is very important for people,

particularly in thinking about, realistically,

if I do this, am I really going to get there?

Tom Chapel, who is an evaluator at CDC,

talks a lot about, “Well, we talk about the

inputs and all that and the outputs, and we talk

about where we want to go. And then, a miracle

happens in the middle.” So, really trying to

think about, what do you really think is going

to happen?

And you can say, okay, this piece I can’t

measure. So, that is a challenge. It may be

that we need to try to think of other ways to

measure it.


Page 13 of 24

Oftentimes, you start with, okay, here’s

how we think this program is going to work. Now

here is what we are actually going to measure.

And so, you keep on going; it is almost a funnel

where you say, okay, here is the whole universe

of what I am talking about, and now I am narrowing

it down.

So, really getting people to talk very

specifically about how they think the program

is going to work. Because I think we tend to

have these assumptions in our heads and don’t

necessarily share them or don’t necessarily put

them down so that anybody can really understand.

Oftentimes, you will find some gaps or sort of

the miracle-happens components.

I really do think it is important to at

least know what you are trying to do. I think

another point that is kind of related is, if you

do this, then, you can get a sense of, if here’s

how you think things are going to happen, if that

piece didn’t happen as you are going along,

well, was it really necessary? Or, whoa, you


Page 14 of 24

had better make sure that that happens because

something went wrong here.

Sometimes you can get to the end and find

that your program didn’t really work. Is it

because you didn’t have enough of it? Or is it

because the logic you thought isn’t really quite

the way behavior change takes place?

This is where theory can help you, too.

If you start with an end, you can think about,

okay, so we want people to be more physically

active. Well, what is involved in that? One

thing is people need to have self-efficacy.

They need to feel they can do it.

The social cognitive theory is one that

is very helpful here because it has got sort of

these attitudinal parts, but also an

environmental and a social component of all the

things that affect whether someone changes

their physical activity behavior.

So, the basic categories that you tend to

see in a logic model – and I am sure all of you

have seen these – the inputs, what you invest,


Page 15 of 24

what you do, what you accomplish in terms of the

events or products, outcomes. It is important

to think about short-, medium-, and long-term,

not just the long-term ones. Because, again,

sometimes given the amount of time you have, you

may only get some of the short-term or the

medium-term done or accomplished or measurable.

And then, the health impact.

I wanted to just talk about a couple

models. One is a template that I found from the

University of Wisconsin Cooperative Extension

that I liked. I think one of the things that

I want to point out is that you can have this

model, and these are the things they handed out.

So, the top sheet, where you do it in columns.

What they did is input. So, that is the

resources, the contributions, the investments.

And then, they have outputs that are

activities and participation. So, the

activities are things like what you do, the

services you provide, the events you have, the

products you have, the promotional activities.


Page 16 of 24

The participation is what I would call exposure

or reach. So, how many people actually attended

a session that you had? How many people did you

reach?

In the models we use that tends to go into

short-term outcomes. And then, the outcomes or

the impact are at the individual level, group

level, community level, whatever level you are

working with, the short-term, medium-term, and

long-term outcomes.

Another way that you can do this is with

a flowchart kind of thing. They also had an

example using an Excel file. It kind of depends

on what works for you. I tend to be a flowchart

kind of woman.

As I said the other day, I take post-its,

and I start with the end and I go, what has to

happen? That allows me to move them around.

Some of the team members have seen me in my

post-it mode.

I think there are a couple of ways that

you can do these, but it is helpful to really


Page 17 of 24

think about where do you want to be in the end,

work back, medium-term, short-term. What are

you actually doing? What are you producing?

And what is available for you to do that?

When you have a logic model, you have got

a lot of stuff to consider. You have got a lot

of potential questions that you could ask. You

have got a lot of steps in the process. And so,

you can’t answer everything. You have to start

really focusing down and really identifying

what is the most important thing for you to do

in your evaluation.

So, in focusing your evaluation, you start

thinking about, No. 1, what is the purpose of

the evaluation? Is it to look at impact for

legislators and the funders? Is to improve your

programs? I mean, sometimes you do an

evaluation; it is more of a pilot, and you are

trying to understand how things work, so that

you can make some changes. So, you need to think

about what is that you are trying to do with this

evaluation.


Page 18 of 24

Who are going to be the users? Again, who

are the stakeholders? What potential uses

might they make of this? So, essentially, what

kinds of things do they want to know? What would

they want to use your evaluation for?

What are your big questions? What

questions do you have about your program? I am

going to talk about questions in two ways. One

is sort of the big evaluation questions, the

research question. The other is the very

specific questions that you might ask on a

survey. So, right now, I am talking about the

big questions.

How are you going to measure it?

Essentially, what are you going to ask? How are

you going to measure it? Who all needs to be

involved and what kinds of agreements do you

need to have in place to do that? So, for

example, do you need to have a data collection

contractor or something like that?

Really, I want to go into a little more

detail on the questions and the methods piece


Page 19 of 24

of that. In terms of the questions, sort of what

big questions do you have about the program?

And what are the questions of highest priority?

So, there is a lot of thinking about what do you

need to know. And then, how are you going to

answer them?

The way people talk about this often is

that the first two are sort of the art of

evaluation, and the last one is the science.

How are you going to do it? What are the methods

that you are going to use?

So, in thinking about what questions you

would have, what are you really going to look

at in your evaluation? Again, you take your

logic model or the model of how you think this

is going to work. If you take, say, this model,

what you can do in thinking about what do you

want to know, you can say, did that step occur?

Did the communication activities happened as

planned? So, essentially, looking at your

activities.

This box here, were relevant


Page 20 of 24

organizations exposed to them? Because, in

this case, this is a program where we are trying

to address people in API organizations. We want

them to take this package and to use it to better

understand NDEP’s API materials and order them.

That is kind of the behavior.

The long-term behavior is to really

incorporate more of NDEP’s materials into their

activities. So, did that step actually take

place? Did we reach them? Did we reach the

people we needed to, and how much did we reach

them? Again, that is kind of looking at one of

these purple boxes.

And then, did awareness of the materials,

requests for, or use of the materials increase?

Again, it is sort of looking at did one of the

steps take place.

The other is looking at the relationships.

Essentially, if someone was reached, if they

were exposed, did it change their awareness?

Really trying to understand sort of some of the

causal stuff.


Page 21 of 24

If we increased awareness, if they really

understood all the materials, did this actually

lead to behavior change or was it associated

with behavior change? That is often what we

want to show.

Often, kind of the big question we have

is, did what we do result in the change that we

wanted? Can we attribute that to what we did,

which is often a very complicated question to

answer?

So, again, we have to think about, and I

think many times we look at, did this happen,

did this happen, did this happen? And we don’t

necessarily think about, if this happened,

based on how I think this is going to work, did

the next step happen, to really understand how

the program actually happened?

If you are really thinking about where do

I want to be, to start thinking about what has

to happen for me to get there. Again, I think

it is kind of related to the piece that you had

about, if it is not measurable, should I put it


Page 22 of 24

in there? It is important for funders, for

legislators, for leadership to really

understand everything that is going to have to

go into changing behavior.

There is a tendency to think that

communication can sometimes do more than it can

or communication with the $1.25 that you have

been given can do what would need a million

dollars. Here is where it helps, I think, to

think about the social/ecological model

because, again, it is not just individual

change. This is why I like it so much. It is

putting puzzles together. It is understanding

how things work or how they might work.

What has happened in some situations is

the funders or the legislators will say, “Well,

we want you to accomplish this in two years, and

we are going to give you $1.25.” If you can show

them that, No. 1, that isn’t going to change in

the amount of time that you have given, and if

you don’t give me more than $1.25, it is really

not going to change.


Page 23 of 24

It is more trying to understand the logic

of everything that has to happen for you to get

the endpoint you want. Now in most of the

programs that I have been involved in, the

program has not been long enough to actually see

the health impact. Two years often is not long

enough to see a change in child mortality due

to diarrhea. But the thing is that, if you can

see a relationship between reducing diarrhea,

reducing complications, then you often will see

there is an association at the scientific level

between not having these conditions and

childhood mortality. So, it is a stretch

because you can’t show it.

But I think it is the same thing with the

National Diabetes Prevention Program. If you

can get someone into the program, if they finish

the program, if they lose the 5 to 7 percent

weight, and they continue, then there is an

association from randomized, controlled trials

that shows that, if they do that, they can

prevent Type 2 diabetes or delay it.


Page 24 of 24

Where is the endpoint and what do we have

to do to get there? Where are places where, if

a partner doesn’t do blah-blah-blah, this isn’t

going to happen? So, again, to be able to

explain why your program didn’t work the way you

wanted, or why it was even better than it could

have been.

That is the value of evaluation. Because

you can, then, speak to why things weren’t

effective and why they didn’t work. If you put

these things in process early on, when you are

planning your whole activity or your total plan,

then you can measure along the way and see what

stopgaps you have or what holes you fell in along

the way.

Documents

Evaluating Health Communication Strategies: Part 1tacenters.emory.edu/resources/products_tools/...A pet peeve of evaluators is, when someone has finished their program and they say,