Metrics and Analytics for new media

New Media Measurementsean zehnder socialesque 09

1Wednesday, April 29, 2009

old-school Media

New-School Interactive

The Interactive Revolution

(How) are video games different?

New‐School Measurement

Outline

www.socialesque.com [email protected] www.socialesque.com/socialesque-blog


old-school Media

Newspaper Audiences

Radio Audiences

TV Audiences

What does each metric imply about the behavior of the audience member -- meaning what are they doing?

What is the smallest unit of an ‘audience’ in each?

How do they size... Why do they size...



May 1996

Portals

May 1997

Virtual Community

Oct 1997

New Measurement

June 1998

New Busines Opportunities

June 1999

New Skills in Demand

July 1999

Disruptive Technologies

Sept 1999

Old Industries In Flux

Aug 1999

Dreaming of Mobile

old-school Media



Most user’s didn’t have broadband

Flash existed, but it was mostly for animations

There was dynamic content, but it required a lot of custom scripting and code to do relatively simple things

Discourse around offline vs. online

Web 1.0

old-school Media



Impressions

Clicks

(later) Time on Screen

Google Analytics “breakthrough” -- finally it is easy to gather and chart basic stats

Data was primarily HTTP/Referrer, Cookie, and JavaScript based

Web 1.0 Measurement

old-school Media



page 61 of July 5, 1993 issue of The New Yorker, (Vol.69 (LXIX) no. 20)

old-school Media



New-School InteractiveApril 2001

Dealing w/ Massive Scale

March 2004

Wisdom of Crowds

Feb 2005

Open Source

Nov 2005

Mobile Revolution

Dec 2006

User-Generated Content

July 2006

Social Networks Emerge

April 2006

Games Grow Up

Aug 2006

Cult of Personality



INTERACTIVE | SOCIAL | MOBILE | UBIQUITOUS | VIRAL | HYBRID




flash | ajax | rails | django | openid | opensocial ...INTERACTIVE | SOCIAL | MOBILE | UBIQUITOUS | VIRAL | HYBRID





2.01 2.85 7.63Miami 1.836.21 2.669.33 7.582.392.08 5.70 5.64

2.47

3.09

4.42

Dec

6.00

Nov

2.86

5.65

2.92

3.65

Oct

3.25

2.88

1.22

2.41

0.20

3.82

Sep

1.94

3.30

1.22

Aug

3.41

0.05

4.22

0.06

3.66

0.85

3.80

July

0.03

3.78

3.65

June

1.59

0.19

3.32

May

1.84

3.80

Apr

3.64

1.37

3.90

2.513.77

3.59

3.06

2.69

Mar

4.03

1.36

3.02

Feb

3.174.35

3.17

1.53

Jan

5.35

San Francisco

Seattle

Chicago

New York

Average Rainfall (inches/month)

You might consider sprucing it up to make it more readable...

Rationale Costs ROI

You believe readability increases the attractiveness of your paper, and attractiveness

will drive sales.

Layout & Graphic Design

More SalesYou believe readability increases the attractiveness of your paper, and attractiveness

will drive sales.


More New Subscriptions

“The Chronicle” 2.0Offline Newspaper

AVERAGE RAINFALL (INCHES/MONTH)

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

San Fran 4.35 3.17 3.06 1.37 0.19 0.11 0.03 0.05 0.20 1.22 2.86 3.09

Seattle 5.35 4.03 3.77 2.51 1.84 1.59 0.85 1.22 1.94 3.25 5.65 6.00

Chicago 1.53 1.36 2.69 3.64 3.32 3.78 3.66 4.22 3.82 2.41 2.92 2.47

New York 3.17 3.02 3.59 3.90 3.80 3.65 3.80 3.41 3.30 2.88 3.65 3.42

Miami 2.01 2.08 2.39 2.85 6.21 9.33 5.70 7.58 7.63 5.64 2.66 1.83

Let’s say you have some data in an newspaper...

Unit Metric

A Sale # of Sales

A Subscription Total SubscriptionsTotal New Subscriptions

“The Chronicle”Offline Newspaper 1.0



Miami

DecNovOctSepAugJulyJuneMayAprMarFebJan

San Francisco

Seattle

Chicago

New York City

4+ 2-3 1-2 <1Average Rainfall (inches/month)

A new online paper opens in your market...

Rationale Costs ROI

They can reach a new audience (e.g. a younger one that gets their info

online).

They hope to learn more about their ‘readers’ by using new metrics.

Layout & Graphic Design Did we reach a

younger demographic?

Can we learn more about our readers

now?


online).

They hope to learn more about their ‘readers’ by using new metrics.

Website creation

Did we reach a younger demographic?


now?


online).

They hope to learn more about their ‘readers’ by using new metrics. Website hosting &

support



now?

theHerald.comOnline Newspaper 1.0

Unit Metric

A Sale # of Sales

A Subscription Total Subscriptions, Total New Subscriptions

An Impression Time On-Screen , # of Impressions

Your notion of an the ‘average reader’ demographic has changed.Your notion of what the ‘average reader’ is DOING has also changed.Note

So you update your own look-and-feel...

“The Chronicle”Offline Newspaper 3.0Rationale Costs ROI

You believe your readership is more interested in getting a sense of how the environment is changing, rather

than using raw rainfall data.

Layout & Graphic Design Did we reach a younger

demographic?

Are we keeping our subscribers or losing them

to the competitor?

You believe your readership is more interested in getting a sense of how the environment is changing, rather

than using raw rainfall data. Updating your printing press to

do color graphics


Are we keeping our subscribers or losing them

to the competitor?




Take-Away: Behind Every Good Metric is a Model of Behavior

San Francisco

Seattle

Chicago

New York

Miami

Average Rainfall (inches/month)

DecNovOctSepAugJulyJuneMayAprMarFebJan

Choose cities...

So your competitor decides to make their graphic interactive...

Unit Metric

A Sale # of Sales

A Subscription Total Subscriptions, Total New Subscriptions

An Impression Time On-Screen , # of Impressions

A Click Most Popular Cities

A Drag Most Popular Months

An Error Number of times the city was changed

A Behavior (generally) Number of Times Shared w/ a Friend

An Event (generally) Number of Times the Server Crashed

theHerald.comOnline Newspaper 2.0

Rationale Costs ROI

They believe that more interactive content is more engaging for their ‘readers.’

They believe that users will share the link to this website with friends if they find the content really

cool (or fun, or useful, etc.).


Can we track how often they share this page with a friend?

Can we track how long they typically spend with the content?

Can we tell if they spend more time with this content than they did reading our static table (from

previous iteration)?




Flash Designer / Scripter







cool (or fun, or useful, etc.). DB Engineer








Hosting & Support








I measure... In order to...

Time

Assess Effectiveness of site on gaining and holding user’s attention

Impressions

Assess Effectiveness of site on gaining and holding user’s attention

System Behavior Save money. Stop bad behavior. Keep system out of user’s way.

User Behavior Ensure they are having good experience. Identify new business opportunities.

More Reasons to Measure





3. Game Simulations enable Designed Experiences

Mechanics - logic and data that makes the system work (e.g. code)

Dynamics - the rules of the game world (e.g. physics)

Aesthetics - the game as experienced/perceived by the user

DynamicsMechanics Aesthetics

1. Games are *highly* interactive

Dynamic Systems

Static SystemsInteractive Systems

2. Games involve simulations

SystemModel Display



(How) are video games different?4. Games require mastery

Perceives the World

Acts in the World

Builds a Theory

Plans Some ActionThe Player (Agent)...

Refines their idea of the world...

5. Since key behaviors differ across games, standardization can be extremely difficult.

Example: “Dead Reckoning”

An expert behavior common in an FPS where a player moves in an arc relative to a target such that

it makes it harder for the target to hit the player while the player is still able to fire on the target.



(How) are video games different?5. Many behaviors are emergent from this complex system.

For one, artificial intelligence is an increasingly important aspect of video games. It is being

used to improve the ‘performances’ of the non-player characters, and to simulate complex

dynamics in real-time such as crowd behavior.

Also, massively multiplayer games have dramatically increased the SCALE of data needing to be measured and processed.

In other words, behavior is emergent.



Eve Online




Eve Online




Eve Online




Eve Online




Eve Online




New-School Measurement

“With Sims Online, we’re trying to take a lot of the community dynamics that we’ve learned from The Sims off-line and reinterpret them in an online world. And we

study the online community all the time. It’s a very interesting community -- it’s over half female, which for

an online game is totally different. And it turns out that a community of 55% females behaves very differently than a community that’s 75% males...In fact we’re capturing

very detailed information. I can tell you how many people are kissing more today than they were yesterday, or how

that’s correlated to other things.” (From Laurel, B. 2003, p. 255).

Will Wright -- Creator of The Sims franchise




Measurement at Massive Scale

Metrics that can’t often be standardized

New ways of thinking about engagement

New reasons to measure

Cross-Media Measurement



Socialesque’s “Rules of Engagement”

What do you expect will be engaging about your application?

What might frustrate users?

What do you hope they do while they use the application?

Did people do what you expected?

Were there things that you notice in the data that you didn’t think to

include in the model?

Which measures are the strongest in your model?

What might fun look like in your game?

What might frustration look like?

What measures might capture engagement?

What is practical to measure from this application?

What can be done to improve the target behaviors?

Can the model of engagement be refined based on the new understanding?

Which items and features were most popular, and might some features need to

be removed or improved to enhance engagement?

analyze refineinstrumentmodel





basic modelexpectationsSocial measures will be most influential.

The effect will be cumulative: the more friends you have lost, the more each loss is felt.

Will have a greater impact with a closer friend.

“A friend leaves the network”

Game Satisfaction & Enjoyment

Amount of Social Interaction

Number of Friends

Likelihood of Attrition

Quality of Friendship

+ + +

-

model

Case Study: Attrition in Social Network



Examination of Context

How often do people play?

How long do people play?

What does a play session look like?

Financial Investment

Brand Loyalty or Favour

Time Investment

USERS

SESSIONS

Social Network

Number of Friends

Quality of Friendship

Network Density

Network Centralization

Network Homogeneity

Aggregate User Behavior

Amount of Interaction

Quality of Interaction

In this example, the user behaviors are

seen as proxies of social network qualities,

since we do not have direct measures for

those variables.

New-School MeasurementCase Study: Attrition in Social Network

model



Survival Time Std. Error 95% Confidence

Interval

Mean 76.32 0.69 74.97, 77.66

Filter QueryStatistical Analysis

Distributed Processing using

‘cloud’ computing

if not in avatars table

random sample

If they don’t have friends

If they never whispered

10 hour query reduced to 30

seconds

11 node Cluster

reduced processing time

by ~100X

analyze




losing a friend *does* increase the chance that a user will leave.

this effect can be reduced when users have strong social networks and

communicate frequently.

however

Regression Analysis of Social Factors in Game Attrition

Predictor B Std. Error Wald Sig. Exp(B)

Model 1Model 1Model 1Model 1Model 1Model 1

Friendship Size –.034 .004 64.381 .000 .966

Cohesiveness –.010 .005 5.229 .022 .990

Outgoing Whispers –.001 .001 5.269 .022 .999

Model 2Model 2Model 2Model 2Model 2Model 2

Friendship Size –.035 .004 66.599 .000 .966

Cohesiveness –.001 .001 4.604 .032 .999

Incoming Whispers –.010 .005 5.208 .022 .990


analyze



Was the data timely enough to effect change?

If users aren’t doing what you expected, should you be trying to redirect them or reward their actual behavior?

After implementing a fix, did you observe the desired outcome?

general

Specific

Additional user-level variables would improve analysis: demographic, account creation/cancellation, etc.

It is possible, from this type of model, to implement a monitoring system to alert Disney of high-risk clusters or even individual users.

The content of the messages can also be an important source of information about the network.

refine




analyze

New-School MeasurementCase Study: Avenue Q Facebook Application



Animated heatmap-style visualization

Contact Sheet for Frame-by-Frame Study

Useful for level balancing & helping users find their way

New-School MeasurementCase Study: Navigation in an FPS

analyze



Pearson Correlations: Navigational Behaviors and Player ReactionsPearson Correlations: Navigational Behaviors and Player ReactionsPearson Correlations: Navigational Behaviors and Player Reactions# Cells Visited

Max Time For A CellDistance Traveled

# Cells VisitedSpeed

# Cells VisitedMax Time For A Cell

Dead Reckoning Count

Freq. of map use .188*Frustration .208*Understanding of Map .194*Understanding of Map .316**Understanding of Map .263**Frustration -.202*Understanding of Map -.311**Intensity -.191***. Correlation is significant at the 0.01 level (2-tailed)**. Correlation is significant at the 0.01 level (2-tailed)*. Correlation is significant at the 0.05 level (2-tailed)*. Correlation is significant at the 0.05 level (2-tailed)

New-School MeasurementCase Study: Navigation + Survey Data in an FPS

analyze



Technology

Metrics and Analytics for new media