…thankstothegeneroussupportof:

2017Celebra,ng10YearsofShapingtheFutureofCorporateCommunica,oninNorway.

10./11.November2016,BI,Oslo

LatestDevelopmentsinEvalua,onofCommunica,onBreakthroughsatLast!

JimMacnamaraPhD,FAMI,CPM,FAMEC,FPRIAProfessorofPublicCommunica@on,UniversityofTechnologySydneyVisi@ngProfessor,LondonSchoolofEconomicsandPoli@calScience

Thispresenta,on

§  Recapsomefundamentals•  Keyprinciples•  Thebasisofevalua@onframeworksandmodels–wheredotheycomefrom?

§  Newframeworksandmodelsforevalua@on•  EuropeanCommissionevalua@onframework•  USTaskForceonstandards•  UKCabinetOfficeevalua@onframework•  AMECIntegratedEvalua@onFramework•  NSWGovernmentEvalua@onFramework

§  Ataxonomyofevalua@on–stages,steps,metrics,andmethodstoapplyinprac,ce§  Casestudies–bestprac@ceevalua@onapplied

•  GoalseRngandmeasurementarefundamentalforcommunica,onandPR

•  Measuringcommunica,onoutcomesisrecommendedversusonlymeasuringoutputs

•  Theeffectonorganiza,onalperformancecanandshouldbemeasuredwherepossible

•  Measurementandevalua,onrequirebothqualita,veandquan,ta,vemethods

•  AVEsarenotthevalueofcommunica,ons

•  Socialmediacanandshouldbemeasured

•  Measurementandevalua,onshouldbetransparent,consistentandvalid

Measurementvs.evalua,on

MEASUREMENT

•  Thetakingofmeasures•  Thecollec@onandanalysisofdatainrela@ontoa

par@cularobject,process,orcondi@on•  SOWHAT?

EVALUATION

•  “Makingajudgement”aboutthevalueorsignificanceofsomething(OxfordandMerriam-Websterdic@onaries)

•  Evalua@onisthesystema@cacquisi@onandassessmentofinforma@ontoprovideusefulfeedbackaboutsomeobject”(Trochim,2006,para,3)

•  “Thesystema@capplica@onofresearchprocedurestounderstandtheconceptualiza@on,design,implementa@on,andu@lityofinterven@ons(Valente,2001,p.106)

•  Assessingresultsagainstobjec,ves

SMARTobjec,ves

§  Containnumbers,percentages,dates

§ Haveameasurementplanforeachstage

§  Berealis,c§  Linkedtoorganisa,onalobjec,vesandgoals§  Achievedbyaspecificdate

Threetypesofevalua,on

Forma@ve

Process

Summa@ve

•  Baseline(e.g.,whatisthecurrentlevelofawareness)

•  Precedents(haveothersimilarprogramsworkedinthepast?)

•  Pre-tes,ng(willwhatweproposework?)

•  CEA(isitthemostcost-effec@veapproach?)

•  Insights/learning(whathasbeenlearnedthatcaninformfuturestrategy,policy,etc?)

•  Impact(whathashappenedasaresultofthecommunica@on?)

• Outcomes–intermediateandlong-term(whatadtudeorbehaviourchangehasoccurred?)

• Monitoringandtrackingoutputs(e.g.,arewereachingtheaudience?)

• Monitoringandtrackingimmediateoutcomes(e.g.,whatresponsearewegedng?

Informa,onprocessing/communica,ontheory

§  Sixstagesofcommunica@on(W.J.McGuire,1968,1969)

ComprehensionPresentation Change/actionRetentionAttention Acceptance

Adver,singmodelsandthe‘marke,ng/salesfunnel’

§  TheAIDAmodelofadver@sing(Strong,1925) AWARENESS

INTEREST

DESIRE

ACTION

Informa,onprocessing/communica,on§  Exposure§  Aben,on§  Interestand/orliking§  Comprehension§  Cogni@on,par@cularlycogni@veelabora@on(thinkingaboutthemessage)§  Acquiringskillsorknowledgerequiredtodealwiththeissue(ifnecessary)§  ARtudechange,par@cularlytoagreement(whatMcGuirecalls“yielding”)§  Storinginforma@oninmemory(reten,on)§  Retrievinginforma@on(i.e.,recall)§  Decidingtoactinaccordancewithinforma@on(inten,on)§  Ac,on/behaviour§  Cogni@veintegra@onofbehaviour(e.g.,reinforcement)§  Encouragingotherstobehavesimilarly–whatMcGuirecalled“prosely@zing”andwhatiscommonly

regardedinmodernmarke@ngas‘advocacy’(McGuire,1999,2001)

ThePIImodelofevalua,on(Cutlip,Center,&Broom,1985)

PyramidmodelofPRresearch(Macnamara,2002,2005)

OUTCOMES(Functional & organisational evaluation)

OUTPUTS(Process & programevaluation)

INPUTS(Formativeresearch)

Number who . change ..…..……………………………………………….. Quantitative surveys (large scale structured)behaviour …...…………………………………………….. Sales; Voting results; Adoption rates; Observation

Number who ….. changeattitudes ……….…………………………………………… Focus groups; Surveys (targeted) (eg Customer, Employee

Number who understand messages..……………………………… Focus groups; Interviews; Complaint decline; ExperimentsNumber who retain messages …….………………………………… Interviews; Focus groups; Mini-surveys; Experiments

Number who consider messages ……....….. ………………………… Response mechanisms (1800, coupons); InquiriesNumber & type of messages reaching target audience ……………………… Media Content Analysis; Communication Audits

Number of messages in the media ……………..….`…………………… Media Monitoring (clippings, tapes, transcripts)Number who received messages ………………...……………………….. Circulations; Event attendances; Web visits & downloads

Number of messages sent …………………………………..……………….. Distribution statistics; Web pages posted

Quality of message presentation …………………………………………………. Expert analysis; Peer review; Feedback; Awards

Appropriateness of message content …………………………………….………….. Feedback; Readability tests (eg. Fog, Flesch); Pre-testing

Appropriateness of the medium selected ………………………………………….…….. Case studies; Feedback; Interviews; Pre-testing (eg. PDFs)

How does target audience prefer to receive information? …………….…………..……… Academic papers; Feedback; Interviews; Focus groupsWhat does target audience know, think, feel? What do they need/want? ………………… Observations; Secondary data; Advisory groups; Chat rooms

& online forums; Databases (eg. Customer complaints)

or Shareholder Satisfaction); Reputation studies

Measurement Methodologies: (formal & informal)

Key Steps/Stages in Communication:

OUT-TAKES(Proposed by some as a 4th stage)

PReffec,venessyards,ck(Lindenmann,1993)

Measuring:BehaviourchangeAttitudechangeOpinionchange

ADVANCED

Measuring:RetentionComprehensionAwarenessReception

INTERMEDIATE

Measuring:TargetaudiencesImpressionsMediaplacements

OUTPUT

LEVEL#3

LEVEL#2

LEVEL#1

PUBLICRLEATIONSEFFECT

IVEN

ESSYARD

STICK

OUTCOMESOUTGROWTHSOUTPUTS

Noble&Watson’sunitedmodel(Noble&Watson,1999)

INPUTSTAGEPlanning&Preparation

OUTPUTSTAGEMessages&Targets

IMPACTSTAGEAwareness&Information

EFFECTSTAGEMotivation&Behaviour

Tacticalfeedback

Managementfeedback

Communica,oncontrolling(DPRG/GPRA,2000;DPRG/IPV,2009)

ResultsofCommunicationProcesses

Output

Outcome

Outflow

InternalOutput

ProcessEfficiencyQualityofWorks

Product

BudgetComplianceThroughputTimesNo.ofShortcomings

Readability/Fogg-IndexSatisfactionofInternal

Clients…

ExternalOutput

MediaCoverageContent

ClippingsVisits

DownloadsImpactRatioShareofVoice

…

DirectOutcome

PerceptionUtil izationKnowledge

AwarenessUniqueVisitorsSessionLengthReaderperIssue

RecallRecognition

…

IndirectOutcome

OpinionAttitudesEmotion

BehavioralDispositionBehavior

ReputationIndexBrandImage

Strategic AwarenessofEmployees

PurchaseIntentionLeads

InnovativeIdeasProject Participation

…

RessourcesEmployeeAssignmentFinancialExpenses

PersonnelCostsOutsourcingCosts

…

Input

ValueCreation

ImpactonStrategicand/orFinancialTargets(ValueChain)Impacton

Tangibleand/orIntangibleAssets

(CapitalAccumulation)

SalesNo.ofProjectAgreements

CostReductionReputationCapital

BrandValueEmployeePerformance

…

Outflow

ORGANISATION

CommunicationProcessesInitiationofCommunicationProcesses

MEDIA/CHANNELS STAKEHOLDERS ORGANISATION

HighImpactonValueCreationWeakInfluenceofCommunicationsManagement

LowImpactonValueCreationStrongInfluenceofCommunicationsManagement

■ The European Commission’s Better Regulation Guidelines uses the terms: ― Inputs ― Outputs ― Results ― Impact (EC, 2015a)

EuropeanCommission(EC,2015a)

10politicalpriorities

Communicationobjectives

Activities Relevance Output Outtake Outcome

Recallindicatorsmeasuringtheextent

towhichthecommunicationactivityislikelytohavesatisfied,

capturedtheattentionofaudiences,orraised

knowledgeandawarenessfocussingontheaudiencedirectlyreached

PoliticalmatchOfthecommunicationactivitiesundertakenwiththe10political

priorities

• Organisingevents• Providing

information• Workingwith

informationnetworksanddesks

Reach indicatorsmeasuringtheextenttowhichextentthecommunicationactivityislikelyto

havereachedtherighttargetaudiencequantitativelyand

qualitatively

EngagementTrustindicatorsmeasuringtheextenttowhichthecommunicationactivityledtoeitheradiscernibleactionbeingtakenorthe

desiredchangeinthetargetaudience’s

perception

InputsResults

Didtheactivitymatchthe10priorities?

Didtheactivityreachthetarget?

Wasthemessagereceived?

Didtheactivitymeettheexpectationsofthetargetaudience?

Didtheactionleadtotheexpectedresult?

DidtheactionchangetheperceptionoftheEU?

Q.

EuropeanCommissionevalua,onmodel(EC,2015b)

PRevalua,onlandscape

§  Inputs/prepara,on§  Ac,vi,es/implementa,on

§  Outputs§  Oubakes§  Outcomes/ouelows/outgrowths

§  Results/effects/impact

“Icouldn’tunderstandwhyPRneededtostandapartfromthedominantapproachtoperformancealignmentandmanagementadoptedbythemajorityoffunc@onsonemightfindinatypicalorganiza@on”

PhilipSheldrakeTheMeasurementStandard,December8,2015,para.6

Programtheoryandprogramlogicmodels

§  Basicprogrammodelevolvedsincethe1970s(KelloggFounda@on,2004,p.1).

Programtheoryandprogramlogicmodels

§  ProgramlogicmodeldevelopedbytheUniversityofWisconsinUniversityCoopera@veExtensionProgram(UWEX)(Taylor-Power&Henert,2008,p.5)

ProgramandPRevalua,onmodels(1985–early2000s)

Basicprogramlogicmod

el–

UWEX

(Taylor-Po

wer&Hen

ert,

2008)

Expa

nded

program

logicmod

el

–UWEX

(Taylor-Po

wer&

Hene

rt,200

8)

ClassicProgramLogicM

odel

(e.g.,KelloggFou

nda,

on

(1998/2004)

Cutlip,Cen

ter&

Broom

(1985)

‘PIImod

el’

Macna

mara’sM

acro

mod

el’(1992)

Lind

enman

n’sE

ffec,vene

ss

Yards,ck(1

993,1997a)

IPRA

GoldPa

pero

nEvalua

,on

(1994,pp.10,18–19)

Fairc

hild(1

997,2001);Fairchild

&O’Con

nor,IPRTo

olkit(1999)

Lind

enman

nStan

dardsa

nd

Guide

lines,199

7b,199

7c)

Nob

le&W

atson’s‘Unifie

dMod

el’(1999)

Grunig&Hon

‘rela,

nships’

mod

el

DPRG

/GPR

A(2000),D

PRG/ICV

(2009)m

odels

LikelyPerform

ance

Measuremen

tFramew

ork

(2000)

Macna

mara’s‘Pryamid’m

odel

(2000,2002a,2002b

)

Lind

enman

n’s

‘guide

lines'(2002/2003)

Input Inputs Inputs Inputs Inputs Inputs(hinted)

Input Input Inputs

Ac@vi@es Planning

Output Outputs•  Ac@vi@es•  Par@cipa@on

Outputs Implement-a@on

Outputs Outputs(basic)

Outputs Output Outputs Output Outputs Output Outputs Outputs PROutputs

Ouqake Ouqakes Ouqakes PROuqakes

Outgrowths

(inter-mediate)

Outcomes Outcomes•  Short-term•  Intermediate•  Long-term

Outcomes Outcomes(advanced)

Outcomes Outcome Outcomes Outcomes Outcome Outcomes Outcomes PROutcomes

Impact Impact Results Business/organiza@onoutcomes

Impact Rela@onships Ourlow Outgrowths Business/organiza@onoutcomes

Effect

9

144

112

51

(Macnamara,2016)

TheoryofChange

INPUTSACTIVITIESOUTPUTSOUTCOMESIMPACT

Communica@on•  Objec@ves•  Targetaudiences•  Strategy

Ac@ons

Short|Medium|Longterm

TheOrganisa@onOrganisa@ongoalsandobjec@ves

Feedbackloopstomonitorandadjuststrategyandtac>csifrequired

ProgramTheory

INPUTSACTIVITIESOUTPUTSOUTCOMESOUTCOMESOUTCOMESShorttermIntermediateLongtermOUTTAKESOUTCOMESIMPACT

Communica@on•  Objec@ves•  Targetaudiences•  Strategy

Feedbackloopstoplanning

Prepara@onProduc@onDistribu@on/Exposure/Recep@onResponseEffectsResultsOrganisa@on/StakeholderImpact

ORGANISATIONOrganisa@ongoalsandobjec@ves

TheoryofChangeProgramTheory

Whatyoudo(COSTCENTRE) Whatyouraudiencedoes(VALUECENTRE)

UKGovernmentCommunica,onServiceevalua,onframework(GCS,2015)

AMECIntegratedEvalua,onFramework(AMEC,2016)

NHSBloodandTransplantblooddona@oncampaign

INPUTSACTIVITIESOUTPUTSOUTCOMESOUTCOMESOUTCOMESShorttermIntermediateLongtermOUTTAKESOUTCOMESIMPACT

Communica@on•  Objec@ves•  Targetaudiences•  Strategy

Feedbackloopstoplanning

Prepara@onProduc@onDistribu@onExposure/Recep@onResponseEffectsResultsOrganisa@on/StakeholderImpact

ORGANISATIONOrganisa@ongoalsandobjec@ves

TheoryofChangeProgramTheory

??????

Taxonomyofevalua,on–6-stagemodel

InputsAc@vi@esOutputsOuqakesOutcomesImpact

TheoryofChange

INPUTSACTIVITIESOUTPUTSOUTCOMESIMPACT

Communica@on•  Objec@ves•  Targetaudiences•  Strategy

Ac@ons

Short|Medium|Longterm

TheOrganisa@onOrganisa@ongoalsandobjec@ves

Feedbackloopstomonitorandadjuststrategyandtac>csifrequired

ProgramTheory

INPUTSACTIVITIESOUTPUTSOUTCOMESIMPACT

TheOrganisa@onOrganisa@ongoalsandobjec@ves

•  Budget•  Personnel•  Exper@se•  Otherresources

•  Paidadver@sing•  Mediapublicity•  Publica@ons(e.g.,

newsleEers,reports)•  Websites•  Socialmediaposts•  Events•  Sponsorships•  Communityprojects

•  Reputa@on•  Rela@onships•  Publicsupport•  Sales/revenue•  Dona@ons•  Costsavings•  Staffreten@on•  Customerreten@on•  Complyingbehaviour

(e.g.,giveblood;drivesafely,getfit)

•  Qualityoflife/well-beingimprovements

Prepara@onProduc@onDistribu@onExposure/Recep@onResponseResults

Affec@veCogni@veCona@veAc@ons

Short|Medium|Longterm

•  Recall•  Awareness•  Interest(e.g.,follows,

likes,retweets)•  Engagement•  Sa@sfac@on•  Adtudechange•  Trust•  Preference•  Inten@ons•  Inquiries/leads•  Advocacy

Feedbackloopstomonitorandadjuststrategyandtac>csifrequired

•  Forma@veresearch•  Crea@vedesign•  Pre-tes@ng•  Planning•  Produc@on•  Mediarela@ons

Communica@on•  Objec@ves•  Targetaudiences•  Strategy TheoryofChange

ProgramTheory

INPUTSACTIVITIESOUTPUTSOUTCOMESIMPACT

TheOrganisa@onOrganisa@ongoalsandobjec@ves

•  Budget•  Personnel•  Exper@se•  Otherresources

•  Forma@veresearch•  Crea@vedesign•  Pre-tes@ng•  Planning•  Produc@on•  Mediarela@ons

•  Paidadver@sing•  Mediapublicity•  Publica@ons(e.g.,

newsleEers,reports)•  Websites•  Socialmediaposts•  Events•  Sponsorships•  Communityprojects

Stakeholders,Publics,Society

Feedbackloopstomonitorandadjuststrategyandtac>csifrequired

•  Sales/revenuedata•  Behaviourtracking•  CRMdata•  Surveys•  Costbenefit/ROI• Well-beingmetrics

•  Surveys(awareness,reputa>on,trust)

•  Interviews•  Socialmediaqualanalysis(likes,shares,comments)

•  NetPromoterScore

• Mediametrics(reach,impressions,OTS,TARPs,CPM)

•  Contentanalysis(e.g.,messages,tone/sen>ment)

• Websitedata•  Socialmediastats•  Abendancefigures•  Readersurveys

•  Pre-surveys•  Focusgroups•  Interviews•  Baselinedatacollec,on(e.g.,databases)

•  Pre-tes,ng•  Peerreview/expertreview

EVALUATIONMETHODS

Prepara@onProduc@onDistribu@onExposure/Recep@onResponseResults

Short|Medium|Longterm

Contextualfactors(e.g.,resourcelimita@ons)

Contextualfactors(e.g.,compe@@vem

essages)

Contextualfactors(e.g.,cogni@vedissonance)

Contextualfactors(e.g.,financialrecession)

•  Recall•  Awareness•  Interest(e.g.,follows,

likes,retweets)•  Engagement•  Sa@sfac@on•  Adtudechange•  Trust•  Preference•  Inten@ons•  Inquiries/leads•  Advocacy

•  Reputa@on•  Rela@onships•  Publicsupport•  Sales/revenue•  Dona@ons•  Costsavings•  Staffreten@on•  Customerreten@on•  Complyingbehaviour

(e.g.,giveblood;drivesafely,getfit)

•  Qualityoflife/well-beingimprovements

Communica@on•  Objec@ves•  Targetaudiences•  Strategy TheoryofChange

ProgramTheory

Measurementandevalua,on(tradi@onalmodel)

Measurement

Evalua@on

Early findings used to fine-tune campaign

Data collection, data analysis Identifying value

Findings primarily used for reporting

TheMAIEModel

Measurement

Early findings used to fine-tune campaign

Inform strategic planning of next stage

INSIGHTS Future business/organisation strategy

Reporting and performance management and review

Data collection, data analysis

Identifying value Evalua@on

1

2 Analysis

Casestudy:Youthan,-cannabiscampaign

Ques,ons&Discussion