Metrics forMMP Development and Operations

Lessons Learned: The Sims Online

Larry Mellon

GDC, Spring 2004

Metrics: Catch-22

Useful

Hard Data

Optimization Tool

Careful…

GI / GO

Expensive

Importance of Metrics is Relative

Lord Kelvin Mark Twain

MeasureEverything

Measure “Just Enough”

Invest No More Than You Need

Kelvin Twain

Complexity

Scale

Precision

$$$ $MMP

MMP: Measurement Focal Points

Infrastructure

Operational Costs

Player Actions

Economy

Similar Use Case: Casinos(Harrah’s “Total Reward”)

Unified Player Action DB

CasinoCasino

Table / MachineTable / MachineTable / Machine

Player Player Player

Track every Player Action

Casino

Highly Profitable, Highly Popular

Unified Player Action DBProfit (per Casino, per Player)

Patterns of Play

Analyze:

Casino Operations

Player Awards Program

Modify:

“This is one of the best investments that we have ever made as a corporation and will prove to forge key new business strategies and opportunities in the future.“ John Boushy (Harrah's CIO, 2000)

TSO: Live Monitors, Summary Views

Player Actions & Persistent Data

Embedded Profiler (Server Side)Automated Report Generators

Designers CommunityManagers

Metrics System

Server Reactions

Engineers Operations

Outline

Background: Metrics & MMP

Implementation Overview

Metrics in TSO

Applications & Sample Charts

WrapupLessons Learned

Conclusions

Questions

Implementation:Driving Requirements

Common Infrastructure

Ease of use

Low overhead

Esper Architecture

esperProbeesperProbeesperProbe

Live Server CPUs

ProcessProcessProcess

UserUserUser

esperLogesperLogesperLog

Event-level Sampling, Aggregated Reporting

esperStore

DBImporter

esperLogesperLogesperLog

esperViewesperViewesperView

esperFetch

t1 t2 t3 t4 t5

esperProbes

Min, Max, Av, Count

Esper Probes• Self-organizing: “class” hierarchy

• Data driven: new probes and/or new game content immediately visible on web

• Example: ESPER_PROBE – (“Object.interaction.%s”, chair->picked)– (“Object.interaction.puppet.%s”, self->picked)

• Human-readable intermediate files

EsperView: Web-Driven Presentation

Daily Reports

Graph Caching & Archiving

Filtering & Meta Data

Report Generator

EsperView: Hierarchical Presentation

Process-Level Collection

Server Cluster

Process Class

Process Instance

Outline

Background: Metrics & MMP

Implementation Overview

Metrics in TSO

Applications & Sample Charts

WrapupLessons Learned

Conclusions

Questions

Applications of Metrics

Load Testing(Realistic Inputs)

Beta Testing & Live Operations (Game Tuning, Community Management)

Load Testing & Live Operations(Server Performance)

Test Servers

Load Testing:“Monkey See / Monkey Do”

AlphaVille Servers

Live Beta Testers

Sim Actions (Player Controlled) Sim Actions (Script Controlled)

Applications of Metrics

Load Testing:Realistic Inputs

Beta Testing & Live Operations: Tuning/Mngmnt

Load Testing & Live Operations:Server Performance

Make Friends, Shake Hands

beats outGive Money / Get Money

Least Used: Disco Dancing

History

Meta Data

Top: TD Dance, WoohooBottom: Dance

Players per Lot 0 to 70 players: < 2 / lot 70 to 400 players: > 3 /

lot

Top: Metrics Bug (sorta)Next: Garden Gnome, Toilet

Bottom: Buffet Table

Beta: numPlayers by numRMates

1. Most have one Roommate

2. Hard-core players: 8

Highest

1. Using Skill Objects2. Using Pizza Maker3. Selling Objects

Lowest

1. Visitor Bonus2. New Avatar3. Object “Rentals”

Economy: Detailed View

1. Using Objects = $$$2. Visitor Bonus == $

Visitor Bonus: Who Makes Money?

1. Most getting no V. Bonus2. Hard-core players: $$

4 of top 5: windows?

?

House Categories (Beta Test)

1. Not a well-used feature2. Shopping least of all

Community Management

Community Actions & Trends

Influencing Player Activity

Free Content

Tracking Problem Players

Marketing

Press Release Teasers

In-Game Brand Exposure

Special Events

NYEve: Kiss Count

                  Esper Cities     All Cities (extrapolated)===========================================New Year's Kiss      32,560             271,333Be Kissed Hotly        7,674             63,950Be Kissed              5,658             47,150Be Kissed Sweetly     2,967             24,725Blow a Kiss            1,639             13,658Be Kissed Hello        1,161              9,675Have Hand Kissed        415              3,458===========================================

Total              52,074        433,949

Applications of Metrics

Load Testing:Realistic Inputs

Beta Testing & Live Operations: Tuning/Mngmnt

Load Testing & Live Operations:Server Performance

DB byte count oscillates out of

control

A single DB Request is clearly

at fault

Most-Used DB Queries (unfiltered)

11,000,000 level Queries need attention, and

drown out others

DB Queries (Filtered)

Filters on 11,000,000 level Queries show patterns of 7,000

level Queries

Incoming & Outgoing Packets

Outline

Background: Metrics & MMP

Implementation Overview

Metrics in TSO

Applications & Sample Charts

WrapupLessons Learned

Conclusions

Questions

Lessons Learned

• Implement early

• Ownership, senior engineers

• Aggregated probes vs event-level tracking

• Automation: collect / summarize / alarms

• “There can be only one”…

Conclusion: Very Useful!

Data MiningOn Players:

Untapped Gold…

Game Design

Realistic Load Testing

Engine Fixes, Optimization

Server Cost, Launch Timing

Critical Feature: Accessibility

Questions

Slides available @

www.maggotranch.com/MMP