Measuring What Matters – Best Practices in PR

Measurement

Katie Delahaye Paine

CEO

Paine Publishing

www.painepublishing.com

@queenofmetrics

measurementqueen@gmail.com

Paine Publishing: Providing communications

professionals the knowledge and information they need to navigate the journey to good measurement

Newsletters

Training

Courses

Consulting painepublishing.co

m

Katie Delahaye Paine: Helping communications professionals define and measure success for 25 years.

Founder of: The Delahaye Group KDPaine & Partners Paine Publishing

Author of: Measuring the Networked

Non-Profit Measure What Matters Measuring Public

Relationships measurementqueen@gmail.com

Don’t Measure What’s Easy

3

Cartoon by Rob Cottingham

Measure What Matters

29/09/2014 News Group International 4

Cartoon by Rob Cottingham

Welcome to the, “I want what

I want, not what you send

me,” era

aka: “The Relationship Era”

®

Where and how people become informed

The old world of communications Today’s Communications

It’s the end of the world as we knew it

The average audience for a MyDrunkKitchen video (http://www.youtube.com/watch?v=uSXQNred3is

1,000,000

CNN average nightly audience 179,000

The number of times per hour Digital Natives switch media—every 2.2

minutes. 27

The percent of conversation that happens OFF LINE 90%

The amount of conversations generated by bots, spammers and pay-

per-click sites 40%

The percent of on-line conversations that are public 10%

The percent of Facebook & Twitter posts that are actually seen < 5%

We need a new Attribution Model

ROI

Other Paid

Marketing

Digital/On-line

Media Buy

Print/TV Media

Buy

ROI

Media Relations

/Social Media/PR

Digital/ Online Media

Buy

Print/TV

Media Buy

Media

Relations/

Social

Media/PR

What’s Changed?

Collapse of mass media

Growth of media

everywhere

Intolerance for messaging

The Barcelona Principles

9

“Viewers are more likely to stop watching commercials at the moment in which brand logos appear on the screen” - ARF Study

The Barcelona Principles, The Conclave &

Industry Standards

1. Importance of Goal Setting and Measurement

2. Measuring the Effect on Outcomes is Preferred to Measuring

Outputs

3. The Effect on Business Results Can and Should Be Measured

Where Possible

4. Media Measurement Requires Quantity and Quality

5. Earned Media Value/AVEs are not the value of Public Relations

6. Social Media Can and Should be Measured

7. Transparency and Replicability are Paramount to Sound

Measurement

All standards are available on

http://www.painepublishing.com/st

andards-central

The Conclave

Standards have been published for:

Content Sourcing & Methods 1

Reach and Impressions 2

Engagement 3

Influence & Relevance 4

Opinion & Advocacy 5

Impact & Value 6

9/29/2014

#SMMStandards – Sources & Methods Transparency Table www.smmstandards.org

Timeframe Analyzed

Research Lead(s)

Channels Analyzed

Data/Content Sources

Analysis Depth ☐ Automated ☐ Manual ☐ Hybrid ☐ All Content Reviewed ☐ Rep. Sample

Source Languages

Search Languages

Sentiment Coding ☐ Automated ☐ Manual ☐ Hybrid ☐ Manual Sampling: _____________________

☐ 3-pt scale ☐ 5-pt scale ☐ Other scale ☐ At entity level ☐ Paragraph/doc level

Spam/Bot Filtering ☐ Automated ☐ Manual ☐ Hybrid ☐ Includes news releases ☐ Excludes releases

Metrics Calculation and Sources

-- Reach

-- Engagement

-- Influence

-- Opinion/Advocacy

Proprietary Methods

Search Parameters See full search string list on page ___ of this report

Transparency Table

#2: Standards for Reach & Impressions

Reach: the scope or range of distribution that a given communication product has in a

targeted audience group; broadcasting, the net unduplicated (also called “deduplicated”)

radio or TV audience for programs or commercials as measured for a specific time

period.

Impressions: Number of people who might have had the opportunity to be exposed to a

story that has appeared in the media (aka opportunity to see (OTS))

Impressions do not equal awareness. Awareness needs to be measured using other

research tools. Impressions are indicative of the OTS. Consider OTS as an alternative

nomenclature to better clarify what impressions really means – [the] potential to see/read.

Multipliers should never be used

OTS must be specific to a particular channel – i.e. For Twitter OTS is the number of first

line followers. For Facebook it is the number of fans to a page.

Eyeball counting

HITS Outcomes

MSM Online Social Media

Impressions are not awareness. Where’s the “So What”?

14

#3: Standards for Engagement

Engagement is an action that happens in response to content – i.e. when someone

engages with you. (not “about you”)

Engagement manifests differently by channel, but is typically measurable based on effort

required and how it is shared with others.

Engagement can be desirable or undesirable

Any measure of engagement must be tied to the goals and objectives for your program

Engagement occurs both off and online, and both must be considered if you intend to

integrate your metrics with other marketing or communications efforts.

Engagement includes such actions as likes, +1, shares, votes, comments, links, links,

retweets, Facebook's “Talking about you” etc.

Engagement actions should be counted by % of the audience that responds.

Likes Are Not Engagement

16

Advocacy Commitment Trial/Consideration

Followers Likes Impressions

9/29/2014

968 1027

914

208

297 261 269

215 222 145

328

204

510

188 222 234

41 71 50 36 25 56 30 71 45

107

Share & Likes Over Time

Average of PostLikes Average of PShares

GWA’s, Photos Drive Shares & Likes

Mia Farrow

David Beckham & Dierks Bentley

Video

• Influence happens when you are persuaded to change behavior or opinion that

would otherwise not have changed.

• Influence can be attributed to either an outlet or an individual.

• Influence happens online and off line and both should be considered

• Should include some combination of the following five elements:

• Reach

• Engagement around individual

• Relevance to topic

• Frequency of posts around the topic

• Audience impact as measured by the ability to get the target audience to

change behavior or opinion.

• If an individual scores a 0 on one element, they have no influence

• Influence cannot be expressed in a single score or algorithm

#4: Influence & Relevance

Big Numbers Influence

All influence is relative

Who is influential depends on your audience

A computer cannot tell you who matters most

To be influential requires relevance, frequency & reach

#5 Standards for Measuring Tone/Sentiment

Whatever process is defined and applied, it must be used

consistently throughout any analysis.

Sentiment reliability varies by vendor and approach – therefore

coding definitions, consistency and transparency are critical.

Opinions, recommendations and other qualitative measures are

typically more valuable than raw sentiment and increasingly

measurable:

Opinions (“it’s a good product”)

Recommendations (“try it” or “avoid it”)

Feeling/Emotions (“That product makes me feel happy”)

Intended action (“I’m going to buy that product tomorrow”)

Testing the accuracy of coding

48.94%

68.31%

88.64%

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

SDL

UberVu

NetBase

% Agreement with human coding

SDL UberVu NetBase

33.12%

58.00%

7.84%

11.95%

26.53%

13.00%

0.32%

9.46%

40.35%

30.00%

92.11%

78.00%

SDL

Beyond

NetBase

UberVu

% positive %negative %neutral

Everyone agrees on Neutral

Not so with Negative and Positive

100.00%

62.32%

14.82%

72.36%

0.00%

37.68%

85.18%

27.64%

Mixed

Negative

Neutral

Positive

Degree to which agreement occurred

No Yes

• Impact and value will always be dependent on client objectives

• Need to define outcomes in advance – will likely span multiple business

goals, especially for social (crosses disciplines)

• “ROI” should be strictly limited to measurable financial impact; “total

value” can be used for financial and non-financial impact combination

• Value can be calculated in positive returns (sales, reputation, etc.) or

avoided negative returns (risk mitigated, costs avoided)

• Key performance indicators and balanced scorecards are helpful to

connect social media impact to business results/language

#6: Impact & Value

ROI

ROI is:

Cost savings

Greater efficiency

Lower legal costs

Faster time to implementation

Lower cost per impact vs. other

marketing channels

Lower cost per message

exposure vs. other channels

ROI is not:

Impressions

Earned media value or AVE

Number of shares, views or

likes

Conversations

$$$$$$$$$$$$$$$$$$$$$$$$$$

Savings, shorter cycles, more

renewals, better ideas,

Product

Support HR IR Sales CI R&D Mktg

Research

Prod.

Mktg

PR//Mkt

25

Social Media isn’t media, it’s your business

9/29/2014

Good Relationships Are More Cost Effective

Type I love Zappos into Google, and you find

1.19 million references

Type Citibank and you get 21,000 references.

Citibank spends 100 times more a year on

advertising than Zappos.

Cost per delegate acquired:

Obama: $6,024

Clinton: $147,058

Romney: $2,389,464

So What?

It’s not about the media, it’s about

your business and your customer

It’s not how loud you’re shouting, it’s

what your stakeholders do after

they’ve heard your message and your

relationships.

There are no boundaries

It’s not about how big your data is, it’s

about how you use it

6 steps to standards-compliant

measurement

Step 1: Define your goal(s).

What outcomes is this strategy or tactic going to achieve?

What are your measurable objectives?

Step 2: Define the parameters

Who are you are trying to reach? How do your efforts

connect with those audiences to achieve the goal.

Step 3: Define your benchmarks.

Who or what are you going to compare your results to?

Step 4: Define your metrics.

What are the indicators to judge your progress?

Step 5: Select your data collection tool(s).

Step 6: Analyze your data.

Turn it into action, measure again

28

Six Steps

to Success

1

2

3

4

5

6

Definitions

Monitoring – process by which data are systematically and regularly collected about a program over time.

Measurement – a way of giving an activity a precise dimension, generally by comparison to some standard; usually done in a quantifiable or numerical manner; see also: data, scale

Metrics – an agreed upon number or percentage that indicates progress towards a goal

Evaluation –a form of research that determines the relative effectiveness of a public relations campaign or program by measuring program*

29

*IPR Dictionary of Measurement Terms, Dr. Donald Stacks www..instituteforpr.org

Why do we communicate?

9/29/2014 30

Outtakes

(Intermediary Effects)

• Awareness

• Knowledge/Education

• Understanding

Outcomes

(Target Audience Action)

• Engagement

• Advocacy

• Revenue/Cost Savings

Activities How does what you do

contribute to the bottom line?

Step 1: Define the goals

What return is expected? – Define in terms of the

mission.

Define your champagne moment. If you are

celebrating complete 100% success a year from

now, what is different about the organization?

31

More than 75% said getting exposure for messages was

the most important goal

Engaging the target audience was the most frequent

definition of success

60.9%

60.9%

60.9%

65.2%

65.2%

65.2%

65.2%

91.3%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

Volume of news coverage

Lack of negative coverage about issues

Quality or number of relationships withcustomers

Outreach to your target audience

Sentiment (positive vs negative) of newscoverage

Change in Behavior/Intent to Act

Quality or number of relationships withstrategic partners

Engaging members of your targetaudience

How do you define success for your organization? Pick all that

apply

3

3

6

0 1 2 3 4 5 6 7

Awareness

Better Relationships

Positive sentiment

What are the top two measures of success for your

organization?

Goals & Suggested Metrics

Goals Goal 1: Meet Sales Targets

Goal 2: Reduce Risk/Threats

Goal 3: Increase market share in

new market

Communications Contribution

• Expand the marketable universe

• Reduce sales cycle

• Increase Trust • Increase

advocacy

• Expand the marketable

universe

Metrics

• % Increase in desirable share of voice

• % in awareness

• % increase in undesirable voice

• % increase in trust scores

• % increase in desirable voice in new market

• % increase in awareness of brand in new market

Step 2: Understand the parameters.

What management’s priorities?

Who are you are trying to reach?

How do your efforts connect with

those audiences to achieve the

goal?

What influences their decisions?

What’s important to them?

What makes them act?

35

Goal: Get the cat to stop howling

Options: Local? Cheap? Convenient?

Strategy: Buy cat food

The Media is the most important stakeholder group for

PRINZ

70% said that National Print/ Online Media was critical

3.2

3.3

3.4

3.5

3.5

3.5

3.8

4.0

4.0

4.3

4.3

4.3

4.4

4.6

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

YouTube

Print advertising

Broadcast advertising

Twitter

Blogs

Facebook

Trade Media

Consumer Media (print or online versions of national magazines)

Friend, family member or colleague

National Broadcast News

Local Print or Online Media

Local Broadcast News

Employees

National Print or Online Media

Keeping in mind your single most important stakeholder group, tell us where you believe they get information? Please rate the following by importance with 5 being most important and 1 being least important.

Step 3: Establish benchmarks

Past Performance Over Time

Think 3

Whatever keeps your

C-suite up at night

38

Comparison to past results is the most common benchmark

for success

Other NZ universities

Other insurance companies

MYOB

Datacom

Other Government agencies

Step 4: Pick your Kick-Butt Index

The Perfect KBI

Is actionable

Is there when you need it

Continuously improves your processes

& gets you where you want to go

You become what you measure, so

pick your KBI carefully

40

The Path to Purchase

High Quality Media Coverage

Intent to support or join

Improved reputation

Legislation passed or killed, new membership

Kick Butt Quality Score

42

Desirable Criteria Score Undesirable Criteria Score

Contains a key message 3.50

No Key Message -2.0

Contains a desirable visual 1.00

Negative message -3.0

Contains a quote from an PRINZ spokesperson 2.50

Contains a competitor or opposition quote

-2.0

Positions PRINZ as a leader 1.00

A story or a headline that leaves the reader less likely to do business with PRINZ

-3.0

The story or headline leaves a reader more likely to do business with PRINZ

2.00

Total 10.00

-10.0

Typical Social Engagement Index

Action Score

Like/Follow/Opens/+1/Favorite .5

Share content (including retweets, forwards, Shares etc.) or shares a link to an ENZ site or page. 2

Signs up to receive email or other content or follows a link to content. 2

Contains one or more of the bank’s key messages .5

Positive Comment (leaves a reader more likely to support ) 1

Contains a quote from an alumni or expert 1.5

Views an video 1

Is from a Tier 1 source 1.5

Total 10

43

Step 5: Pick the right measurement tools

If you want to measure messaging, positioning,

themes, sentiment:

Content analysis

If you want to measure awareness, perception,

relationships, preference:

Survey research

If you want to measure engagement, action,

purchase:

Web analytics

If you want predictions and correlations

you need two out of three

All the tools you really need

Google Analytics

Ignore page views

Set up conversions

Excel

9/29/2014

64% wish they had access to Industry Engagement

More than 85% have access to Press Clippings.

Social Analytics is the least available tool.

Step 5: Selecting a measurement tool

48

Objective KBI Tool

Increase message

communications

Increase percent of items containing one or

more messages

Media Content Analysis

Increase

awareness/preference

% of audience preferring your brand

to the competition

Survey Monkey, Survata

Engage

marketplace

% increase in engagement on website and/or

social sites

Simply Measured, Google

Analytics, Site Catalyst,

Network Analysis

Components of a Relationship Index Control mutuality

In dealing with people like me, this organization has a tendency to throw its weight around. (Reversed)

This organization really listens to what people like me have to say.

Trust This organization can be relied on to keep its promises.

This organization has the ability to accomplish what it says it will do.

Satisfaction Generally speaking, I am pleased with the relationship this organization has established with people like me.

Most people enjoy dealing with this organization.

Commitment There is a long-lasting bond between this organization and people like me.

Compared to other organizations, I value my relationship with this organization more

Exchange relationship Even though people like me have had a relationship with this organization for a long time; it still expects

something in return whenever it offers us a favor.

This organization will compromise with people like me when it knows that it will gain something.

This organization takes care of people who are likely to reward the organization.

Communal relationship This organization is very concerned about the welfare of people like me.

I I think that this organization succeeds by stepping on other people. (Reversed)

How to implement relationship metrics

Step 1: Conduct a benchmark relationship

study

Step 2: Implement PR program

Step 3: Conduct a follow up relationship

study

Step 4: Look at what’s changed

Step 6: Be Data Informed, not Data Driven

Rank order results from worst to best

Ask “So What?” at least three times

Put your data into an overall framework consistent with C-Suite expectations

Find your “Data Geek”

Compare to last month, last quarter, 13-month average

51 Page 51

Photo Event

High Message Content

Resource Use

Low

Hig

h

Med

ium

Ver

y h

igh

Level of Engagement

Ver

y H

igh

Med

ium

Hig

h

Low

High Resources

No Message

Low Resources

Webinar

Status update

Link

Ultimate Road Trip

Google + Chat

Media Day

Corporate Video

Advocacy vs. resource use

Resource Use

Low

Hig

h

Me

diu

m

Ve

ry h

igh

Ver

y H

igh

Med

ium

Hig

h

Low

High Resources Low Resources

Webinar

Status update

Link

Ultimate Road Trip

Google + Chat

Media Day

Corporate Video

Real-Time Metrics focus on Outcomes

53

PUBLIC AFFAIRS & BRAND DASHBOARD Priority 1: Brand Awareness Priority 2: Advocacy Priority 3: New Membership Culture

Bench-mark

Current Variance Bench-mark

Current Variance Bench-mark

Current Variance Bench-mark

Current Variance

% increase in share of desirable voice

8.9% 2.17% ↓6.73 pts

% increase in PRINZ share of Desirable Conversation around Digital privacy

10% 15 ↑6.73 pts

% Increase in lawyers who are extremely or very familiar with PRINZ’s positioning

38% 38% Annual Data Only

% increase in ratings on communications trustworthiness (from culture survey)

54% New data

available July 14

N/A

% decrease in share of undesirable voice

1% 1% Un-changed

% decrease in Government officials unaware of PRINZs position on Digital Privacy

10% 10% ↑6.73 pts

% Increase in online engagement on issue

25% 0 1% % increase in employee knowledge of PRINZ “Way We Work” (From Knowledge Quizzes)

23% New data

available Q3

N/A

% increase in awareness among lawyers (from 2014 Brand Metrics Survey)

46% 46% Next data

available Q3 14

% increase in Inbound info requests for info

516(Apr) 427 (May)

↓17%

% employees strongly understanding strategic objectives (From Knowledge Quizzes)

32% New data

available Q3

N/A

% increase in perception of PRINZ as a leader in Digital Privacy

5% 5% Un-changed

% increase in engagement on PRINZ.org.nz

202K

197K

↓2%s

% reduction in Say/Do gap on agility metrics (average of agile & make decisions faster)

36.5% New data

available Q4

N/A

Below Target

At Risk

On Target

Connecting data yield greater insights

106 38 24

73 101 49 25

1087

1331

699 637

848

1331

699

10.25

35.03

29.13

8.73 8.40

27.16 27.96

0

5

10

15

20

25

30

35

40

0

200

400

600

800

1000

1200

1400

Jan Feb Mar April May June July

Relationship between interviews and placements

Number of Interviews Number of placements Interview per placement

ACA has made a demonstrable difference in the quality of

media coverage

If an ACA or one of its

programs was mentioned

in a media story, it was:

More likely to contain

an endorsement

More likely to be

positive

More likely to contain

key messages

Less likely to be

negative

0.22

0.00

0.62

1.42

4.61

0.08

0.08

0.01

-0.61

-0.66

Endorsements

Positive Visual

Negative Visual

Tone

MessageCommunication

Average of OCS

Differences in Quality of Coverage with and without

ACA

Atlantic City Atlantic City Alliance

Without ACA events, OCS Scores for Atlantic City would

have been significantly lower

4

3.25

2.75 2.99

3.65

2.96 3.36 3.24

2.34 2.37 2.43

1.30

-1.24

0.37 -0.05

0.28 0.28

-1.56

4.91

3.92

2.99

3.58

4.14 4.1 4.27 4.12 4.29

2.78 2.56

1.53

-0.29

0.61 0.20

1.44 1.77

0.63

-2

-1

0

1

2

3

4

5

6

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June

Ave

rage

OC

S Sc

ore

Average OCS Score Over Time

Atlantic City OCS without ACA Atlantic City OCS with ACA

The red line represents coverage of Atlantic City minus all mentions of ACA and its programs

Free entertainment generated the highest OCS Scores in

Q2

5.26

5.73

6.36

6.54

6.71

7.15

7.31

7.36

7.50

7.67

8.00

8.10

8.68

9.30

Miss America

Meet AC

DO AC

July 4th Fireworks

Air Show

Blake Shelton

Miss'd America Pageant

Sand Blast

Hello Summer

Boardwalk Hall Light Show

Challenge Triathlon

Sand Sculpting World Cup

Lady Antebellum Concert

Free Entertainment

Top Programs By OCS

When ACA programs received media coverage, traffic

followed

0

50

100

150

200

250

300

0

20000

40000

60000

80000

100000

120000

140000

Relationship between ACA Program Mentions and Site Visits

Site Visits Program Mentions

ACA programs drive higher OCS scores, which correlate

highly with web visits

0.41

0.44

0.47

AC items

ACA Items

ACA OCS Scores

Correlations between Web Visits and PR Metrics

Pearson r. value

White House Volunteerism Office (CNCS) is able to connect

specific social outreach to registrations on serve.gov

61

The ROI of Emily – So What?

62

Red line

indicates

media

impressions

35,152,789 OTS

So What = Revenue

63

10 Ways to find the Money for Measurement

1. Don’t call it measurement – It’s Research

2. Consolidate - Social? IR? Agencies? Research?

3. Crowdsource – Who else has tools or surveys?

4. Spread the cost over 2 fiscal years

5. Reach out to universities

6. Take advantage of free stuff

7. Monitor only what matters

8. Narrow your Top Tier media list to what you CAN afford

9. Google analytics

10. Facebook Insights

11. Twitter Analytics

Remember These Points

It’s not about the media, it’s about the business and the customer 1

It’s not about Big Data, but about how you use it. 2

You need to be data informed, not data-driven 3

It’s not how loud you’re shouting it’s about relationships. 4

Standards are a reality not an excuse to hide behind 5

Thank You!

For more information on measurement, read my blog:

http://www.painepublishing.com/blog

For a copy of this presentation or to subscribe to our newsletter, give me

your card or email me at measurementqueen@gmail.com

Follow me on Twitter: @queenofmetrics

Friend me on Facebook: Katie Paine

Or call me at 1-603-682-0735

66