72
Opening the Black Box

Game Analytics: Opening the Black Box

Embed Size (px)

DESCRIPTION

This presentation outlines the basics of game analytics and the symbiotic relationship between game analytics and game user research, and provides 10 examples of cool game analytics and game data mining methods applied across the board of indie to AAA titles.

Citation preview

Page 1: Game Analytics: Opening the Black Box

Opening the Black Box

Page 2: Game Analytics: Opening the Black Box

� In one decade: 200M -> 2B consumers� Global consumers

� Across demographics

� Business models have changed -> � More online

� Less retail

� F2P and micro-transactions

� Games across multiple devices

Page 3: Game Analytics: Opening the Black Box

Games are now big data:

� Billions of monthly game sessions

� Hundreds of millions of active users

� Hundreds of TBs captured daily

� Exponential increase in data flow

Page 4: Game Analytics: Opening the Black Box

Big Game Data

New /old business models

State machines

Context data

Persistence

Page 5: Game Analytics: Opening the Black Box

� Games are state machines

� During play, a person generates an ongoing

loop of actions and system responses that

keep the game state changing

� Compared to e.g. web shops, games

generate a lot more actions and responses

State machines

Page 6: Game Analytics: Opening the Black Box

� New business models require analytics to drive revenue

� Other business models must adopt analytics

� More data and new methods

New business models

Retail business

model

Page 7: Game Analytics: Opening the Black Box

� Games are more persistent/played for longer

� Users have more devices

� More and new ways to play

� Scaling up of GUR data collection online

Persistence

Page 8: Game Analytics: Opening the Black Box

� Not just gameplay data: 360 degree view on users� Profile data from social networks

� Off-game social networks

� In-game social networks

� Micro-transactions

� Advertising

� Payment systems

� Game system behavior/responses

� Geolocation

� Psychographic marketing

� …

Context data

Page 9: Game Analytics: Opening the Black Box

"You are no longer an individual,

you are a data cluster bound to a

vast global network"

- Watch Dogs

Page 10: Game Analytics: Opening the Black Box

� Data surge also at small scales

� More data sources for GUR work

� If you have a game, you can get deep data about users -> data collection is (sort of) democratic� Middleware tools enable tracking

� Mobile phones enable geolocation

� Facebook data from users

� Payment data

� ...

Page 11: Game Analytics: Opening the Black Box
Page 12: Game Analytics: Opening the Black Box
Page 13: Game Analytics: Opening the Black Box

� Methods - game analytics - remains vague and require expertise

� Accessible mostly to large companies

� Methods infancy compared to other industries

� Minimal available knowledge/-exchange

� Models are often not explained to the end user of analytics results

Page 14: Game Analytics: Opening the Black Box
Page 15: Game Analytics: Opening the Black Box

So …. what it is Game Analytics anyway?

Page 16: Game Analytics: Opening the Black Box

� Business intelligence (BI)

� A set of theories, methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information (Wikipedia – pretty accurate )

� Aim of BI: to provide support for decision-making at all levels of an organization� Making an organization data-driven

Page 17: Game Analytics: Opening the Black Box
Page 18: Game Analytics: Opening the Black Box

� Analytics is the process of BI

� Analytics is the process of discovering and

communicating patterns in data towards

solving problems in business

� Supporting decision management

� Driving action

� Improving performance

� - Or for purely frivolous and artistic reasons!

Page 19: Game Analytics: Opening the Black Box

� A specific domain of analytics: game

development and game research

� The game as a product: user experience,

behavior, revenue …

� The game as a project: the process of

developing the game

Page 20: Game Analytics: Opening the Black Box

� Data source for most product game analytics

� Any amount of quantitative, unprocessed data obtained over any distance, which pertain to game development or game research.

� Describes attributes about objects

� Many sources: Installed clients, game servers, mobile units, user testing/playtesting

Page 21: Game Analytics: Opening the Black Box

� Metric = Interpretable, quantitative measure of one or more attributes of one or more objects –operating in the context of games

� Object: virtual item, player, user, process, developer, forum post ....

� Attribute: an aspect of the object

� Context: tied to process, performance or users of games.

Page 22: Game Analytics: Opening the Black Box

� Telemetry data from Quake III� Data on the location of the player avatar

� Weapons used, hit/miss information, etc.

� Convert raw attribute data about players into features� E.g. “number of hits” or “number of misses”

� Domain = 0-1000 (with 1000 being max. no of hits)

� Calculate across 1+ features to develop game metrics� E.g. hit/miss ratio

� Weapon use times

� Weapon accuracy

� Player accuracy

Page 23: Game Analytics: Opening the Black Box

� A game metric does not need to be derived

from telemetry data.

� “average completion time” for a game level

� Stopwatch ->

� Telemetry ->

� - but with different measurement accuracies

both are metrics

Page 24: Game Analytics: Opening the Black Box

� User metrics

� Metrics related to the users, viewing them as either customers (revenue sources) or players(behaviors)

� ARPU, DAU, MAU -> customer focus

� Avg. Playtime, Completion Rate -> player focus

� Focus on revenue streams and user experience

Page 25: Game Analytics: Opening the Black Box

User

Vastly most common data source!

Page 26: Game Analytics: Opening the Black Box

� Performance metrics

� Measures of the performance of technical

infrastructure and software execution

� FPS, load balancing …

� Patch impacts

� Bug counts, bug ratios …

� Heavily used by Quality Assurance

Page 27: Game Analytics: Opening the Black Box

� Process metrics

� Related to the process of developing games

� Game development is a creative process -> can be

hard to monitor/quantify

� Today numerous tools, e.g. task size estimation

tools, burn down charts

� Focus is on measuring and qualifying production

Page 28: Game Analytics: Opening the Black Box

� Metrics that are unrelated to the games context = business metrics

� the revenue of a game development company last year

� the number of employee complaints last month

� Usually project or company focused

� Useful to integrate with game metrics

Page 29: Game Analytics: Opening the Black Box

The Knowledge Discovery Process in GA

Page 30: Game Analytics: Opening the Black Box

Attribute definition

Data acquisition

Data pre-processing

Metrics development

Analysis and evaluation

Visualization

Reporting

Knowledge deployment

GA Knowledge Discovery Cycle

Page 31: Game Analytics: Opening the Black Box

� Strategic GA: The global view on how a game should evolve based on analysis of user behavior and the business model.

� Changing a monetization model

� Tactical GA: informs game design at the short-term,

� A/B test of a new game feature.

� Operational GA: Analysis and evaluation of the immediate, current situation in the game.

� Removing a bug, adapting game to user behavior in real-time.

Page 32: Game Analytics: Opening the Black Box

� Common methods in game analytics:

� Description

� Statistics

� Data Mining

� Machine Learning/AI

� Programming

� Operations Research

� Visualization

Page 33: Game Analytics: Opening the Black Box
Page 34: Game Analytics: Opening the Black Box

Operationalizing behavioral telemetry from games

Page 35: Game Analytics: Opening the Black Box

� How many players do we have?

� What is the percentage of jumps cleared?

� What is the ratio of new to leaving players?

(churn)

� How many rocket launchers were fired on

average per multiplayer game?

Page 36: Game Analytics: Opening the Black Box

� KISS …

0

100000

200000

300000

400000

500000

600000

700000

800000

900000

1000000

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37

Su

m c

red

its

ea

rne

d p

er

lev

el

Financial Index

Archer

Berserker

Lancer

Mystic

Priest

Slayer

Sorcerer

Warrior

Page 37: Game Analytics: Opening the Black Box

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

%D

IFF

Min/Max Difference - Financial Index

Page 38: Game Analytics: Opening the Black Box

0

0.5

1

1.5

2

2.5

3

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

Archer

Berserker

Lancer

Mystic

Priest

Slayer

Sorcerer

Warrior

Page 39: Game Analytics: Opening the Black Box

� What activities to players engage in for how

long? Does this match design intent?

(DeR

osa,

Gam

asutr

a)

Page 40: Game Analytics: Opening the Black Box

� What behaviors, entities or objects go together?

� Do players open door 2 and take the east route? Or the west route?

� Do players buy the Red Pet with the Blue Pet – or the Orange Pet?

� Do players with the Sword of Mayhem +5 do better against Dr. EvilBoss than those with the Olive Branch of Peace -5?

Page 41: Game Analytics: Opening the Black Box

� Association (affinity): when performing an

association analysis, the goal is to find

features (attributes) that “go together”

� Defines association rules in the data: If X,

then Y

� “if players buy Striped Trousers of Strength +3,

they will also buy Girdle of Charisma +2.”

Page 42: Game Analytics: Opening the Black Box

� Girdle of Charisma +2

� Striped Trousers of

Strength +5

Association rules have measures of support and

confidence

Support: the frequency of features occurring

Confidence: the probability that when X appears,

so does Y

Page 43: Game Analytics: Opening the Black Box

� What is the rate of progression through the map?

� Are there areas/points where progression is halted?

� (below left from Halo 3, © Bungie)

Page 44: Game Analytics: Opening the Black Box

How likely is this player to convert to a paying

user?

What is the Life Time Value of this player?

How likely is this player to join a guild?

How likely is this player to reach the endgame?

Page 45: Game Analytics: Opening the Black Box

Based on known values we want to

predict possible future values

Model developed based on training data

Widely applied!

Page 46: Game Analytics: Opening the Black Box
Page 47: Game Analytics: Opening the Black Box

� Which decision chains leads a user to become a paying users?

� What makes a player quit?

� Decision tree algorithms attempt to find relationships between input values and target values

� Allow us to follow the effect of player decisions, and plan for how to promote specific decisions

Page 48: Game Analytics: Opening the Black Box

At what level will a player stop as a feature of playtime and rewards in Tomb Raider: Underworld?

Level-2 Rewards Rewards > 10

Level-3 playtime-> playtime > 43 minutes : 4 -> playtime < 43 minutes : 7

Rewards < 10 : 2

Page 49: Game Analytics: Opening the Black Box

� Where do we loose players?

� Which levels make people take long breaks?

� Which upgrade paths do people choose?

Page 50: Game Analytics: Opening the Black Box

� An analysis of flow through a series of actions

� Tracking loss of players during signup

� Evaluating the steps leading up to a purchase

� Drop-offs in player retention

Page 51: Game Analytics: Opening the Black Box

(Ubisoft, Square Enix)

Page 52: Game Analytics: Opening the Black Box

� Deathmap (“heatmap”)

� Killmap

� Balance map

� Resource use map

� Performance map

� Player density map

� Purchase map

� Quitmap

� …

Heatmaps

Page 53: Game Analytics: Opening the Black Box
Page 54: Game Analytics: Opening the Black Box
Page 55: Game Analytics: Opening the Black Box
Page 56: Game Analytics: Opening the Black Box

� Games are experienced spatio-temporally

� All games require movement

� All games take time to play

� Why is analytics then mainly temporal?

Page 57: Game Analytics: Opening the Black Box

� Spatio-temporal analytics� Does not reduce the dimensions of game metrics

data

� Deals with the actual dimensions of play.

(Image: Ubisoft)

Page 58: Game Analytics: Opening the Black Box

� Decades of knowledge in spatial analytics outside of games – ripe for harvesting� Trajectory analysis (how do users play the game?

Move in 3D?)

� Spatial outlier detection (finding exploitation spots, bugs)

� Spatial clustering (are players distributed across maps? Or is some content not used?)

� Spatial co-location patterns/trends (analysis of army composition in RTS)

Page 59: Game Analytics: Opening the Black Box
Page 60: Game Analytics: Opening the Black Box

� How do people play the game?

� Which ways of playing are the most popular?

� How can we detect outliers? (extreme groups)

� How do we find the factors most important to

characterize user behaviors?

Page 61: Game Analytics: Opening the Black Box

� Behavioral telemetry can be high-dimensional� 100’s of player actions

� Clustering is used for reducing dimensionality and finding commonalities

� Goal: group/segment objects so that intra-cluster similarity is high and interclustersimilarity is low.

Page 62: Game Analytics: Opening the Black Box

SIVM: finding extreme profiles

� Assassins� Veterans � Target dummies

� Assault-Recon� Medic-Engineer � Driver � Assault wannabee

Page 63: Game Analytics: Opening the Black Box

� Each different playstyles, and different things that keep them in the game

� ”Driver”: drives, flies, sails – all the time and favors maps with vehicles

� ”Assassin”: kills – afar or close – no vehicles!

� ”Target dummies”: unskilled newbies – high dropout unless they quickly transfer to another cluster

Page 64: Game Analytics: Opening the Black Box

� Use clustering to evaluate if there is sufficient

variety in gameplay

� Use behavioral clustering to find profiles,

then cater to them – in real-time!

� Monitor players´ profiles to track behavior

changes: target dummy -> veteran

� Useful for evaluating learning curves fx.

Page 65: Game Analytics: Opening the Black Box
Page 66: Game Analytics: Opening the Black Box

Game Analytics Game User Research

Page 67: Game Analytics: Opening the Black Box

� GUR is about the user and how they function when dealing with the software (Ben Lewis-Evans). � “Small” scale, in-depth

� GA is the process of discovering and communicating patterns in data towards solving problems in business � ”Large scale”, more shallow

� Both seek to inform decision making

Page 68: Game Analytics: Opening the Black Box

GA GURU

sers

Qu

ali

tati

ve m

eth

od

s (

?)

Process

Business

Performance

Page 69: Game Analytics: Opening the Black Box

Experience

(playtesting)

Usability

(user testing)

Behavior

(analytics)

Larger scale

Smaller scale

Page 70: Game Analytics: Opening the Black Box
Page 71: Game Analytics: Opening the Black Box

� Handout: an (mostly) online reference list� Books + tools in the list have info on how to perform

the 10 analyses mentioned (!)

� Recommended: RapidMiner, WEKA

� ”Getting Started with Game Analytics”� Series introducing GA to the non-expert

� Blog.gameanalytics.com

� Contact: [email protected]

Page 72: Game Analytics: Opening the Black Box