Jon puleston main stage - 2013

The Festival of NewMR 2013 would not be possible without our sponsors. Thanks to:

Gold Sponsors

Silver Sponsors

Session Sponsors

Media PartnerFestival Supporters

• Q – Research Software• CreaMetrix Research• TapestryWorks• Keen as Mustard• CrowdLab

• L&E Research• Schlesinger Associates• Focus Forums• eCGlobal Solutions• Revelation

Jon Puleston, GMI Interactive, UK

Festival of NewMR 2013 - Main Stage

REINVENTING THE 5 STAR

FILM RATING PROCESSJon Puleston VP Innovation GMI Interactive

****”It was great”

****“Totally Awesome”

****”Brilliant”

****”Amazing!!!!!”



THIS RESEARCH IS A WORK IN PROGRESS

• I am exploring this topic

• As such I would like you to view this from a

critical standpoint

• I am keen to know what you think

• Open to any ideas to help me move forward



MY AMBITION IS TO DESIGN A

BETTER FILM RATING SYSTEM

WHY?





NEARLY EVERY FILM SEEMS

TO SCORE 4 STARS!



5% 5%

30%

40%

20%

0%

5%

10%

15%

20%

25%

30%

35%

40%

1 2 3 4 5

DISTRIBUTION OF IMDA FILM REVIEW SCORES

WE EFFECTIVELY

DON’T USE



2nd REASON:

THIS ISSUE IS REPRESENTATIVE OF THE

WIDER ISSUE WE FACE WITH ALL

RANGE SCALES USED IN SURVEYSThey tend to deliver badly differentiated data



1 2 3 4 5

THE AVERAGE* DISTRIBUTION OF

ANSWERS ON A 5 POINT LIKERT SCALE

= slightly agree

= quite like

* This chart is based on the consolidated answers to 650 questions



BRAND ASSOCIATIONS: 5 POINT SCALE RATING

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

IT LEADS TO FLAT DATA THAT LOOKS LIKE THIS…



THINKING ABOUT WHY MOST

FILMS SCORE 4 STARS?

STARTING POINT….



WHY DO MOST FILMS SCORE 4?

NON RESPONSE BIAS: • We mostly only watch films we think we will like

• We are potentially more motivated to rate films we enjoy



RATING OF A FILM

COMPLICATED PIECE OF THINKING




DID YOU ENJOY THE FILM?

• Yes/no = 90% in under 0.5 SECOND

• Answers are instinctive

HOW WOULD YOU RATE IT?

• Giving a 5 star rating = 2-10 SECOND

• Answers not instinctive

• Complex and subjective



BECOME A SOCIAL LANGUAGE

I ENJOYED IT

IT WAS A LETDOWN FOR ME

WARNING

I ENDORSE THIS

WE ENJOY 80% OF FILMS WE SEE




ONE OTHER ISSUE…

RATING NATURALLY PROMOTES

ANALYTICAL THINKING?

• That elevates intellectual judgements over emotional

reaction for some people (particularly men!)





0 2 4 6 8 10

Sta

r ra

tin

g

Well madeEnjoyment



HO

W W

EL

L M

AD

E

HOW MUCH ENJOYED

ROMCOMS

FOREIGN FILMS

ART HOUSE

COMEDIES

THRILLERS

DRAMAS

BLOCK BUSTERS

HORROR

B MOVIES

MUSICALS

* Note this is an exaggerated scale for illustrative purposes only

WHY ROMCOMS DON’T WIN OSCARS?



SO IN SUMMARY

• Most films we see we are likely to like them

• The complexity of the thinking process prompts a

group to ignore the subtleties of the task and just

say I enjoyed it

• Scores are a social language and “enjoyed it” =4

• Another group judge it in an analytical/ intellectual

dimension and discount their real feelings

The result = meaningless homogenised data



SO HOW COULD WE CREATE

A MORE EFFECTIVE RATING

PROTOCOL?



STARTS WITH HAVING A THINK ABOUT

WHAT CONSUMERS WANT*

• Something that is simple and easy to interpret

• Has some context i.e. 4 stars compared to what?

• Takes account of genre: e.g. a thriller/romcom/art house

• Can be trusted

• Takes account of peoples individual taste & preferences

• Provides an assessment of the risk of whether you will enjoy it

• Provides an understanding of what a film will offer you – clues as to how

we can enjoy a film

• Pragmatic but not patronising

* Based on 20 qualitative in depth interviews with various film lovers including my

Mum, girlfriend, daughter and the 3 people I sit next to in the office! (i.e. I am not

saying this this is wholly comprehensive and objective feedback!)



LOOKING AT ALTERNATIVE APPROACHES FROM A

MARKET RESEARCH PERSPECTIVE WHAT WE HAVE

BEEN EXPLORING ARE 5 AVENUES:

1. Using more descriptive scales

2. More creative questioning approaches

3. Alternative judgement dimensions

4. Exploring comparative evaluation techniques

5. A Linguistic scoring system



USING MORE DESCRIPTIVE SCALES



LEADS TO WONKY ANSWER DISTRIBUTION

FOUND NO SATISFYING SOLUTION



GREATQUITE

GOODBRILLIANTTV

MOVIEINSPIRING

GOT LOST IN CHOICES – MESSY DATA

BORING FUNNY

EXPLORED A MULTI CHOICE EVALUTION PROCESS



EXPERIMENTED WITH A RANGE OF

MORE CREATIVE SCORING TECHNIQUES



GOOD FOR REDUCING PATTERN ANSWERING IN

REPETITIVE QUESTIONS

0

0.05

0.1

0.15

0.2

0.25

0.3

1 2 3 4 5 6 7 8 9 10

Balance of data

Heart Grid

0%

5%

10%

15%

20%

25%

30%

35%

40%

1 2 3 4 5 6 7

Grid format Interactive format

Patter answers

BUT THAT IS NOT REALLY THE PROBLEM TRYING TO SOLVE IN THIS CASE



EXPLORED ALTERNATIVE MEASUREMENT DIMENSIONS

JUST AS FLAT DATA: 70% OF FILMS MET PEOPLES EXPECTATIONS



ALTERNATIVE MEASUREMENT DIMENSIONS

DIFFICULT TO DETERMINE WHAT THIS MEANT



CHOICE BASED ASSESSMENT



CHOICE BASED ASSESSEMENT



TRADITIONAL MONADIC STAR RATING



Pleas note: this can only be treated at anecdotal evidence based on small pilot

CHOICE BASED

DIFFERENTIATING BUT COMPLEX



SELF ANCHORED CHOICE COMPARISONS

WAS GRAVITY BETTER OR WORSE THAN SPACE ODYSSEY

NICE BUT STILL QUITE COMPLICATED!



GATHERED 10,000 FILM REVIEWS ACROSS 8 ENGLISH SPEAKING COUNTRIES

LINGUISTIC ANALYSIS



ANALYSED THE CHARACTER OF FEEDBACK*

COMPLAINTS

DAMNING REMARKS

BASIC PRAISE

SUPERLATIVES

ACCOLADES

e.g. Fantastic

e.g. The best

HOW IT MADE ME FEEL

HOW IT MADE ME THINK

OBSERVATIONS

NONCHALANCE

e.g. I enjoyed/Exciting

Scary/Funny/Sad etc

e.g. Thought provoking

e.g. It was good

e.g. Well Directed

e.g. Not bad

e.g. Slow

e.g. Rubbish

Highest scoring

Lowest scoring

* Based on the analysis of 10,000 short films reviews



Enthusiometer

For market researchers of quality & distinction



DEVELOPING A WORD SCORING SYSTEMa system based upon respondents’ ratings of words

super

great

gem

thrilling

fascinating

inspiring

exciting

very good

beautiful

clever

entertaining

engrossing

enjoyable

enchanting

innovative

classic

amusing

hilarious

moving

interesting

adventurous

touching

good

fun

cool

incredible

pretty good

heart-warming

very nice

emotional

liked

provoking

quite good

decent

quirky

different

fine

nice

bizarre

average

typical

ordinary

confusing

light weight

childish

juvenile

dated

overrated

dull

not great

nothing

poor

disliked

boring

mindless

disappointing

stupid

exceptional

masterpiece

spectacular

fantastic

outstanding

superb

excellent

amazing

greatest

brilliant

awesome

wonderful

fabulous

epic

powerful

favourite

loved

marvellous

best

absurd

bad

very poor

nonsense

pointless

useless

lousy

sucked

drivel

pathetic

dreadful

garbage

junk

awful

rubbish

terrible

crap

the worst

disgrace

4 points

3 points

3 points

2 points

2 points

1 points

-2.5 points

-2.5 points

-0.5 points

-3 points

“extremely ****”

“really ****”

“quite ****”

“incredibly ****”

multiply by 1.75

multiply by 1.5

multiply by 0.5

multiply by 2

“not ****”

multiply by -0.5



0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

1 3 5 7 9 1113151719212325272931333537394143454749515355

5 star scorring system language scoring system

This is how the natural language scoring

system correlates with actual ratings…



AUGMENTED WITH A NET PROMOTER STYLE APPROACH

SUPERLATIVES COMPLIMENTS MEDOCRITY NEGATIVES

amazingawesome

brilliantclassic

excellentpowerful

greatinteresting

lovebeautiful

satisfy

goodlike

niceokayfine

all right

annoybad

boringcrap

overratedrubbish

% of

POSITIVE

WORDS

subtract

% of

NEGATIVE

WORDS



LINGUISTIC NET PROMOTER DELIVERED FAR

GREATER DIFFERENTIATION OF SENTIMENT



“good sci-fi story with excellent special effects”

WHAT STOOD

OUT ABOUT IT

WHAT TYPE OF

FILM IT WAS

“it was hilarious. i loved the minions and couldn't

stop smiling”

“great film, most enjoyable whilst playing

my emotions”

“some excellent action, a reasonable plot

although a little transparent”

AN OPEN-ENDED REVEALS MORE THAN

JUST SENTIMENT & RATINGS



INFORMATION WE COULD MINE FROM THE COMMENTS

CEREBRAL IMPACT e.g. challenging

AESTETICS e.g. Cool, Beautiful

ACTING/DIRECTING/STORY e.g. great acting

ETHUSIASM e.g. Fantastic

LOVE/HATE e.g. Loved it/it was OK

MEETING EXPECTATION e.g. Not as bad as a I thought

CHAT VALUE e.g. Number of words written

CONNECTION LEVEL e.g. I enjoyed v it was enjoyable

EMOTIONAL IMPACT e.g. heart warming

HUMOUR e.g. Hilarious

PERSONALITIES e.g. Love De Niro



BEGAN TO CODE & CLUSTER COMMENTS BY THEMES

HUMOUR

“funny”

“hilarious”

“humorous”

“made me

laugh”

“comical”

“light-

hearted”

ACTION

“action”

“action-

packed”

“lively”

“explosive”

“exciting”

“high-

octane”

DRAMA

“emotional”

“romantic”

“heart-

warming”

“touching”

“inspiring”

“love”

“dramatic”

TENSION

“tense”

“thrilling”

“jump”

“suspenseful”

“scary”

“dark”

“intense”

“anxious”

THEMATIC DIFFERENCES



EMOTIVE COGNATIVEVS

“emotional”

“heart-

warming”

“touching”

“happy”

“love”

“joy”

“interesting”

“educational”

“thought-

provoking”

“engaging”

“moralising”

“values”

BEGAN TO CODE & CLUSTER COMMENTS BY THEMES



Emotion

93%

Reason

7%

Emotion

29%

Reason

71%

Twilight Breaking Dawn – Part 2 The Hunger Games

BECOMES A DISCRIPTIVE TOOL



A DESCRIPTIVE TOOLsee where films fits in

120% MORE

STORY*

148% MORE

ACTION*150%MORE

ACTING SKILL*

63%MORE

DIRECTION & STYLE*

174% MORE

STORY*

* COMPARED TO A TYPICAL ACTION MOVIE



A DESCRIPTIVE TOOLsee where films fits in

81% MORE

DRAMA*

82%MORE

HUMOUR*

159% MORE

TENSION*

200%MORE

VISUAL APPEAL*

118%

MORE

TENSION

* COMPARED TO A TYPICAL COMEDY * COMPARED TO A TYPICAL DRAMA



SUMMARY OF WHAT WE HAVE LEARNT

• Realise that we are not quite there yet

• What we realised is that the challenge is not to

substitute this measure but enhance it and make it

more meaningful through better contextualisation

• What is actually needed is a more multi dimensional

measure

• This needs to be simple and universally easy to

understand



NEXT STEPS

• Rolling out thinking to different categories & topics

• Doing some experiments in different languages

• Working on developing some Infographics to

communicate a more multidimensional measure

» We then plan to test these out on cinema goers to see

what they find most useful

• Preparing a proper paper on all the learnings



Q & A

Jon Puleston

GMI Interactive

Lenny Murphy

GreenBook



Jon Puleston

Email: [email protected]

TWITTER @JONPULESTONwww.question-science.blogspot.co.uk

Marketing

Jon puleston main stage - 2013