Scalability Tools: Automated Testing

(30 minutes)

Overview

Hooking up your game external tools internal game changes

Applications & Gotchas engineering, QA, operations production & management

Summary & Questions

Review: controlled tests & actionable results useful for many purposes

(1)

Repeatable tests, using N synchronized game clients

(2)

High-level, actionablereports for many audiences

Programmer Development Director Executive

Test Game

Automated testing accelerates large-scale game development & helps predictability

TimeTargetLaunch

ProjectStart

TSO case study: developer efficiency

Strong test supportWeak test support

Oops

Ship Date

Time

% Complete

autoTest

Better game

earlier

Measurable targets & projected trends give you actionable progress metrics, early enough to react

Any test(e.g. # clients)

Target

Time

Any Time(e.g. Alpha)

First PassingTest

Now

Oops

Success stories

Many game teams work with automated testing EA, Microsoft, any MMO, …

Automated testing has many highly successful applications outside of game development

Caveat: there are many ways to fail…

How to succeed

Plan for testing early Non-trivial system Architectural implications

Fast, cheap test coverage is a major change in production, be willing to adapt your processes Make sure the entire team is on board Deeper integration leads to greater value

Kearneyism: “make it easier to use than not to use”

Automated testing components

Test ManagerTest ManagerTest Selection/SetupControl N Clients

RT probes

Any Game

Startup&

Control

Scriptable Test Client(s)Scriptable Test Client(s)Emulated User Play Session(s)

Multi-client synchronization

Repeatable, Sync’edTest I/O

Report ManagerReport ManagerRaw Data Collection

Aggregation / SummarizationAlarm Triggers

Collection&

Analysis

Input systems for automated testing

algorithmic recordersscripted

Game code

Multiple test applications are required, but each input type differs in value per application. Scripting gives the best

coverage.

Hierarchical automated testing

unit systemsubsystem

• Faster ways to work with each level of code• Incremental testing avoids noise & speeds defect isolation

Multiple levels of testing gives you

Input (Scripted Test Clients)

Command steps

…Validation steps

…

Pseudo-code script of users play the game, and what the game should do in response

createAvatar [sam]enterLevel 99buyObject knifeattack [opponent]

checkAvatar [sam exists]checkLevel 99 [loaded]checkInventory [knife]checkDamage [opponent]

Test Client (Null View) Game Client

Scripted Players: Implementation

Script Engine

State

Game GUI

Game Logic

Commands

State

Presentation Layer

Or, load both

Test Client

Test-specific input & output via a data-driven test client gives maximum flexibility

Regression Load

InputAPI

ReusableScripts & Data

OutputAPI

Script-SpecificLogs & Metrics

Key Game StatesPass/Fail

Responsiveness

A Presentation Layer is often unique to a game

NullView Client

Some automation scripts should read just like QA test scripts for your game

TSO examples routeAvatar, useObject buyLot, enterLot socialInteraction (makeFriends, chat, …)

Input (data sets)

Mock data

Repeatable tests in development, faster load,

edge conditions

Unpredictable user elementfinds different bugsReal data

RandomEdge cases & real world

performance

Repeatable Debugging & benchmarking

Common Gotchas Not designing for testability

Retrofitting is expensive Blowing the implementation

Brittle code Addressing perceived needs, not real needs

Use automated testing incorrectly Testing the wrong thing @ the wrong time Not integrating with your processes Poor testing methodology

Testing the wrong time at the wrong time

C o d eC o m p le tio n

T im e

A lp h a

D es ig nS p ac e

T im e

A lp h a

Applying detailed testing while the game design is still shifting and the code is still incomplete introduces

noise and the need to keep re-writing tests

Build Acceptance Tests (BAT) Stabilize the critical path for your team

Keep people working by keeping critical things from breaking

Final Acceptance Tests (FAT) Detailed tests to measure progress against milestones “Is the game done yet?” tests need to be phased in

More gotchas: poor testing methodology & tools Case 1: recorders

Load & regression were needed; not understanding maintenance cost

Case 2: completely invalid test procedures Distorted view of what really worked (GIGO)

Case 3: poor implementation planning Limited usage (nature of tests led to high test

cost & programming skill required) Case 4: not adapting development

processes Common theme: no senior engineering

analysis committed to the testing problem

Automated Testing for Online Games

Overview

Hooking up your game external tools internal game changes

Applications engineering, QA, operations production & management

Summary & Questions

“The difference between us and a computer is that the computer is blindingly stupid, but it is capable of being stupid many, many millions of times a second.”

Douglas Adams (1997 SCO Forum)

Repeat massive numbers of simple, easily measurable tasks

Mine the results Do all the above, in

parallel, for rapid iteration

Automated testing: strengths

Semi-automated testing is best for game development

TestingRequirements

Rote work (“does door108 still open?”)

Scale Repeatability Accuracy Parallelism

Integrate the two for best impact

Automation Creative bug

hunting, visuals Judgment calls,

playability Reacting to change, Evaluating autoTest

results

Manual Testing

Plan your attack with stakeholders(retire risk early: QA, Production, Management)

Tough shipping requirements (e.g.) Scale, reliability Regression costs

Development risk Cost / risk of engineering & debugging Impact on content creation

Management risk Schedule predictability & visibility

Automation focus areas (Larry’s “top 5”)

Performance Scale is hard to get right

Critical path stability Keep team going forward

Non-determinism Gets in the way of everything

Content regression Massive, recurring $$

Compatibility & install Improves life for you & user

Yikes, that all sounds very expensive!

Yes, but remember, the alternative costs are higher and do not always work

Costs of QA for a 6 player game – you need at least 6 testers at the same time

Testers Consoles, TVs and disks & network Non-determinism

MMO regression costs: yikes2

10s to 100s of testers 10 year code life cycle Constant release iterations

Stability: keep the team working!(TSO use case: critical path analysis)

Failures on the Critical Path block access to

much of the gameenter_house ()

Test Case: Can an Avatar Sit in a Chair?

use_object ()

buy_object ()

buy_house ()

create_avatar ()

login ()

Checkin

Development

Prevent critical path code breaks that take down your team

Sniff Test

Pass / fail, diagnostics

Candidate code

Safe code

Stability & non-determinism (monkey tests)

Code Repository Compilers

Continual Repetition of Critical Path Unit Tests

Reference Servers

AutoTest addresses non-determinism

Detection & reproduction of race condition defects Even low probability errors are exposed

with sufficient testing (random, structured, load, aging)

Measurability of race condition defects Occurs x% of the time, over 400x test

runs

Monkey test: enterLot ()

Monkey test: 3 * enterLot ()

Four different behaviors in thirty runs!

Content testing (areas)

Regression Error detection Balancing / tuning

This topic is a tutorial in and of itself Content regression is a huge cost problem Many ways to automate it (algorithmic, scripted

& combined, …) Differs wildly across game genres

Content testing (more examples)

Light mapping, shadow detection Asset correctness / sameness Compatibility testing Armor / damage Class balances Validating against old userData … (unique to each game)

Automated Testing for Online Games

(One Hour)Overview

Hooking up your game external tools internal game changes

Applications engineering, QA, operations production & management

Summary & Questions

Summary: automated testing

Start early & make it easy to use Strongly impacts your success

The bigger & more complex your game, the more automated testing you need

You need commitment across the team Engineering, QA, management, content

creation

Q&A & other resources My email: larry.mellon_@_emergent.net More material on automated testing for games

http://www.maggotranch.com/mmp.html Last year’s online engineering slides This year’s slides Talks on automated testing & scaling the development

process www.amazon.com: “Massively Multiplayer Game

Development II” Chapters on automated testing and automated metrics

systems www.gamasutra.com: Dag Frommhold, Fabian Röken

Lengthy article on applying automated testing in games Microsoft: various groups & writings

From outside the gaming world Kent Beck: anything on test-driven development http://www.martinfowler.com/articles/continuousIntegratio

n.html#id108619: Continual integration testing

Amazon & Google: inside & outside our industry