Moving Communication Forward With Evaluation

Jim Macnamara PhD, FAMI, CPM, FAMEC, FPRIA

Professor of Public Communication, University of Technology Sydney Visiting Professor, London School of Economics and Political Science

Measurement & evaluation Measuring communication and ‘big data’!

• Goal setting and measurement are fundamental for communication and PR

• Measuring communication outcomes is recommended versus only measuring outputs

• The effect on organizational performance can and should be measured where possible

• Measurement and evaluation require both qualitative and quantitative methods

• AVEs are not the value of communications

• Social media can and should be measured

• Measurement and evaluation should be transparent, consistent and valid

SMART objectives

Contain numbers, percentages, dates

Design evaluation at the planning stage

Be realistic

Linked to organisational objectives and goals

Achieved by a specific date

Three types of evaluation

Formative

Process

Summative

• Baseline (e.g., what is the current level of awareness)

• Precedents (have other similar programs worked in the past?)

• Pre-testing (will what we propose work?)

• CEA (is it the most cost-effective approach?)

• Insights / learning (what has been learned that can inform future strategy, policy, etc?)

• Impact (what has happened as a result of the communication?)

• Outcomes – intermediate and long-term (what attitude or behaviour change has occurred?)

• Monitoring and tracking outputs (e.g., are we reaching the audience?)

• Monitoring and tracking immediate outcomes (e.g., what response are we getting?

Information processing / communication theory

Six stages of communication (W. J. McGuire, 1968, 1969)

ComprehensionPresentation Change/actionRetentionAttention Acceptance

Advertising models and the ‘marketing/sales funnel’

The AIDA model of advertising (Strong, 1925)

AWARENESS

INTEREST

DESIRE

ACTION

Information processing / communication

Exposure

Attention

Interest and/or liking

Comprehension

Cognition, particularly cognitive elaboration (thinking about the message)

Acquiring skills or knowledge required to deal with the issue (if necessary)

Attitude change, particularly to agreement (what McGuire calls “yielding”)

Storing information in memory (retention)

Retrieving information (i.e., recall)

Deciding to act in accordance with information (intention)

Action/behaviour

Cognitive integration of behaviour (e.g., reinforcement)

Encouraging others to behave similarly – what McGuire called “proselytizing” and what is commonly regarded in modern marketing as ‘advocacy’ (McGuire, 1999, 2001)

The PII model of evaluation (Cutlip, Center, & Broom, 1985)

Macro model of evaluation (Macnamara, 1992)

Pyramid model of PR research (Macnamara, 2002, 2005)

OUTCOMES(Functional &

organisational evaluation)

OUTPUTS(Process & program

evaluation)

INPUTS(Formative

research)

Number who . change ..…..……………………………………………….. Quantitative surveys (large scale structured)

behaviour …...…………………………………………….. Sales; Voting results; Adoption rates; Observation

Number who ….. change

attitudes ……….…………………………………………… Focus groups; Surveys (targeted) (eg Customer, Employee

Number who understand messages..……………………………… Focus groups; Interviews; Complaint decline; Experiments

Number who retain messages …….………………………………… Interviews; Focus groups; Mini-surveys; Experiments

Number who consider messages ……....….. ………………………… Response mechanisms (1800, coupons); Inquiries

Number & type of messages reaching target audience ……………………… Media Content Analysis; Communication Audits

Number of messages in the media ……………..….`…………………… Media Monitoring (clippings, tapes, transcripts)

Number who received messages ………………...……………………….. Circulations; Event attendances; Web visits & downloads

Number of messages sent …………………………………..……………….. Distribution statistics; Web pages posted

Quality of message presentation …………………………………………………. Expert analysis; Peer review; Feedback; Awards

Appropriateness of message content …………………………………….………….. Feedback; Readability tests (eg. Fog, Flesch); Pre-testing

Appropriateness of the medium selected ………………………………………….…….. Case studies; Feedback; Interviews; Pre-testing (eg. PDFs)

How does target audience prefer to receive information? …………….…………..……… Academic papers; Feedback; Interviews; Focus groups

What does target audience know, think, feel? What do they need/want? ………………… Observations; Secondary data; Advisory groups; Chat rooms

& online forums; Databases (eg. Customer complaints)

or Shareholder Satisfaction); Reputation studies

Measurement Methodologies: (formal & informal)

Key Steps/Stages in Communication:

OUT-TAKES(Proposed by some as a 4th stage)

PR effectiveness yardstick (Lindenmann, 1993)

Measuring:Behaviour changeAttitude changeOpinion change

ADVANCED

Measuring:RetentionComprehensionAwarenessReception

INTERMEDIATE

Measuring:Target audiencesImpressionsMedia placements

OUTPUT

LEVEL #3

LEVEL #2

LEVEL #1

PU

BLI

C R

LEAT

ION

S EF

FEC

TIV

ENES

S YA

RD

STIC

K

OUTCOMES OUTGROWTHS OUTPUTS

Noble & Watson’s united model (Noble & Watson, 1999)

INPUT STAGEPlanning & Preparation

OUTPUT STAGEMessages & Targets

IMPACT STAGEAwareness & Information

EFFECT STAGEMotivation & Behaviour

Tactical feedback

Management feedback

Communication controlling (DPRG/GPRA, 2000; DPRG/IPV, 2009)

■ The European Commission’s (EC) Better Regulation Guidelines uses the terms:

―Inputs

―Outputs

―Results

―Impact

■ European Commission model (EC, 2015b)

European Commission (EC, 2015a, 2015b)

10 political priorities

Communication objectives

Activities Relevance Output Outtake Outcome

Recall indicators measuring the extent

to which the communication

activity is likely to have satisfied,

captured the attention of audiences, or raised

knowledge and awareness focussing

on the audience directly reached

Political match Of the communication activities undertaken with the 10 political

priorities

• Organising events • Providing

information • Working with

information networks and desks

Reach indicators measuring the extent to which extent the

communication activity is likely to

have reached the right target audience

quantitatively and qualitatively

Engagement Trust indicators measuring the extent to which the communication

activity led to either a discernible action being taken or the

desired change in the target audience’s

perception

Inputs Results

Did the activity match the 10 priorities?

Did the activity reach the target?

Was the message received? Did the activity meet the expectations of the target audience?

Did the action lead to the expected result? Did the action change the perception of the EU?

Q.

PR and communication evaluation landscape

Inputs / preparation

Activities / implementation

Outputs

Outtakes

Outcomes / outflows / outgrowths

Results / effects / impact

Program theory and program logic models

Basic program model evolved since the 1970s (Kellogg Foundation, 2004, p. 1).

Program theory and program logic models

Program logic model developed by the University of Wisconsin University Cooperative Extension Program (UWEX) (Taylor-Power & Henert, 2008, p. 5)

Program and PR evaluation models (1985–early 2000s)

Bas

ic p

rogr

am lo

gic

mo

de

l –

UW

EX (

Tayl

or-

Po

we

r &

He

ne

rt,

20

08

)

Exp

and

ed

pro

gram

logi

c m

od

el

– U

WEX

(Ta

ylo

r-P

ow

er

&

He

ne

rt, 2

00

8)

Cla

ssic

Pro

gram

Lo

gic

Mo

de

l (e

.g.,

Ke

llogg

Fo

un

dat

ion

(1

99

8/2

00

4)

Cu

tlip

, C

en

ter

& B

roo

m (

19

85

) ‘P

II m

od

el’

Mac

nam

ara’

s M

acro

mo

de

l’ (1

99

2)

Lin

de

nm

ann

’s E

ffe

ctiv

en

ess

Y

ard

stic

k (1

99

3, 1

99

7a)

IPR

A G

old

Pap

er

on

Eva

luat

ion

(1

99

4, p

p.

10

, 18

–19

)

Fair

child

(1

99

7, 2

00

1);

Fai

rch

ild

& O

’Co

nn

or,

IP

R T

oo

lkit

( 1

99

9)

Lin

de

nm

ann

Sta

nd

ard

s an

d

Gu

ide

line

s, 1

99

7b

, 19

97

c)

No

ble

& W

atso

n’s

‘U

nif

ied

M

od

el’

(19

99

)

Gru

nig

& H

on

‘re

lati

nsh

ips’

m

od

el

DP

RG

/ G

PR

A (

20

00

), D

PR

G/I

CV

(2

00

9) m

od

els

Like

ly P

erf

orm

ance

M

eas

ure

me

nt

Fram

ew

ork

(2

00

0)

Mac

nam

ara’

s ‘P

ryam

id’

mo

de

l (2

00

0, 2

00

2a,

20

02

b)

Lin

de

nm

ann

’s ‘

guid

elin

es'

(2

00

2/2

00

3)

Input Inputs Inputs Inputs Inputs Inputs (hinted)

Input Input Inputs

Activities Planning

Output Outputs

Activities Participation

Outputs Implement-ation

Outputs Outputs (basic)

Outputs Output Outputs Output Outputs Output Outputs Outputs PR Outputs

Outtake Outtakes Outtakes PR Outtakes

Outgrowths

(inter-mediate)

Outcomes Outcomes Short-term Intermediate Long-term

Outcomes Outcomes (advanced)

Outcomes Outcome Outcomes Outcomes Outcome Outcomes Outcomes PR Outcomes

Impact Impact Results Business / organization outcomes

Impact Relationships Outflow Outgrowths Business / organization outcomes

Effect

9

14

4

1

12

5 1

(Macnamara, 2016)

Theory of Change

INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT

Communication • Objectives • Target audiences • Strategy

Actions

Short | Medium | Long term

The Organisation Organisation goals and objectives

Feedback loops to monitor and adjust strategy and tactics if required

Program Theory

INPUTS ACTIVITIES OUTPUTS OUTCOMES OUTCOMES OUTCOMES Short term Intermediate Long term OUTTAKES OUTCOMES IMPACT

Communication • Objectives • Target audiences • Strategy

Feedback loops to planning

Preparation Production Distribution /Exposure/Reception Response Effects Results Organisation/Stakeholder Impact

ORGANISATION Organisation goals and objectives

Theory of Change Program Theory

What you do (COST CENTRE) What your audience does (VALUE CENTRE)

UK Government Communication Service evaluation framework (GCS, 2015)

AMEC Integrated Evaluation Framework (AMEC, 2016)

NHS Blood and Transplant blood donation campaign

INPUTS ACTIVITIES OUTPUTS OUTCOMES OUTCOMES OUTCOMES Short term Intermediate Long term OUTTAKES OUTCOMES IMPACT

Communication • Objectives • Target audiences • Strategy

Feedback loops to planning

Preparation Production Distribution Exposure/Reception Response Effects Results Organisation/Stakeholder Impact

ORGANISATION Organisation goals and objectives

Theory of Change Program Theory

? ? ? ? ? ?

Taxonomy of evaluation – 6-stage model

Inputs Activities Outputs Outtakes Outcomes Impact

Evaluation taxonomy includes …

Low cost and even no-cost evaluation methods as well as advanced methods

Start small and work up – e.g., pilot studies

Learn from failures as well as successes – learnings inform future strategy

Visualise findings

Busy management want ‘dashboards’, simple charts

1. Not accessing data that are available Literature review ‘OP’ data Unrecognised data – look beyond numbers to consultation submissions, correspondence, etc.

2. Lack of data sharing

Breaking down ‘data siloes’ and politics – Knowledge management (KM) expertise

3. Lack of data analysis – tools and skills

Qualitative data analysis methods such as textual, content, and thematic analysis

4. Lack of resources and rigour Academic partners Interns

5. The output to impact gap – how to bridge it?

Build an evaluation bridge using a multi-stage, multi-step framework/model

Tips to overcome 5 barriers and move evaluation forward

Questions & Discussion