28
REAL GOOD UNREAL TESTING CHARLES P. SCHULTZ Author “Game Testing All In One” INDIE GAMES SUMMIT

Unreal conference slides

Embed Size (px)

DESCRIPTION

Slides from today's Indie Games Summit at FIU

Citation preview

Page 1: Unreal conference slides

REAL GOODUNREAL TESTING

CHARLES P. SCHULTZAuthor “Game Testing All In One”

INDIE GAMES SUMMIT

Page 2: Unreal conference slides

TEST SUBJECTS

WHAT CAN I TEST?• UNREAL ENGINE VERSIONS, PORTS• MODS• LEVELS• WEAPONS, VEHICLES• PC, CONSOLE or MOBILE

INDIE GAMES SUMMIT

Page 3: Unreal conference slides

TEST OBJECTIVES

• FIND BUGS• PINPOINT CAUSES

- Environment- Repeatability- Frequency

• ESTABLISH WHAT WORKS• VERIFY FIXED BUGS• GIVE DEVS FEEDBACK

INDIE GAMES SUMMIT

Page 4: Unreal conference slides

TESTER’S WEAPONS

• PARAMETERS

• MODELS

• USE CASES

• METHODS

• TOOLS

• PLAYER STYLES- Achiever, Explorer, Killer, Socializer

INDIE GAMES SUMMIT

Page 5: Unreal conference slides

BOUNDARY VALUES

• 0• MINIMUM• MAXIMUM• MAXIMUM + 1• SPECIAL VALUES

- High Scores- February 29th

- Zone Boundaries: KOTOR example

INDIE GAMES SUMMIT

Page 6: Unreal conference slides

GAME PHASES

PRE-GAMEPRE-GAME

ACTIVATE

GAME START

Initialize system componentsLoad maps and game assetsEstablish network communicationGame controllers detectedetc.

IN-GAME POST-GAME

DEACTIVATE

INDIE GAMES SUMMIT

Page 7: Unreal conference slides

DEFECT TRIGGERS

• NORMAL BEHAVIOR

• CONFIGURATION

• STARTUP

• EXCEPTIONS

• STRESS

• RESTART

INDIE GAMES SUMMIT

Page 8: Unreal conference slides

DEFECT TRIGGERS

• NORMAL BEHAVIOR

- typically 80 to 100% of tests

- most costly to find

- relatively low cost to repair

- less effective as project progresses

- serious issues removed before ship

INDIE GAMES SUMMIT

Page 9: Unreal conference slides

DEFECT TRIGGERS

• CONFIGURATION• STARTUP• EXCEPTIONS• STRESS• RESTART

- up to 20% of tests- higher rate of defects found per test- requirements may be lacking

INDIE GAMES SUMMIT

Page 10: Unreal conference slides

TRIGGER GOALS

• NORMAL BEHAVIOR: 40%• CONFIGURATION: 20%• STARTUP : 10%• EXCEPTIONS: 20%• STRESS: 5%• RESTART: 5%

- grow non-Normal test cases- elicit requirements from developers

INDIE GAMES SUMMIT

Page 11: Unreal conference slides

RESTART

“You can fast swap weapons to reload the first one to make it fire faster, for example you can shoot the sniper, quickly swap to another weapon lets say hammer, swap back and shoot the sniper again as the reload time has been wiped”

BUG EXAMPLE

INDIE GAMES SUMMIT

Page 12: Unreal conference slides

COMBINATORIAL TABLES

• SELECT PARAMETERS TO TEST- GUN, AMMO, RANGE

• SELECT TEST VALUES FOREACH PARAMETER- AMMO: 0, 1, MAXIMUM

• PAIR EACH VALUE WITH ANOTHERPARAMETER’S VALUES

- AMMO=1 withGUN=ROCKET, LINK GUN, SNIPER

INDIE GAMES SUMMIT

Page 13: Unreal conference slides

COMBINATORIAL TABLES

• START WITH FIRST VALUE OF ONE PARAMETER

GUN AMMO RANGE

ROCKETROCKETROCKET

01

MAX

1MAX

MAX+1

INDIE GAMES SUMMIT

Page 14: Unreal conference slides

COMBINATORIAL TABLES

• FOR THE NEXT PARAMETER, CREATE NEW PAIRS

GUN AMMO RANGE

ROCKETROCKETROCKET

01

MAX

1MAX

MAX+1LINK GUNLINK GUNLINK GUN

01

MAX

MAXMAX+1

1

INDIE GAMES SUMMIT

Page 15: Unreal conference slides

COMBINATORIAL TABLES

• CONTINUE UNTIL ALL PAIRS ARE SATISFIED

GUN AMMO RANGE

ROCKETROCKETROCKET

01

MAX

1MAX

MAX+1LINK GUNLINK GUNLINK GUN

01

MAX

MAXMAX+1

1SNIPERSNIPERSNIPER

0

MAX1

MAX+11

MAX

INDIE GAMES SUMMIT

Page 16: Unreal conference slides

COMBINATORIALNOMICS

• For our 3x3 matrix, we covered all pair combinations with only 9 tests. A set of 13 parameters with 3 values would require 1.5 million tests for all combinations, but only requires 15 tests for pairs

• FOR LARGE OR COMPLEX SETS OF PARAMETERS, USE A TOOL

ALLPAIRShttp://www.satisfice.com/tools.shtml

INDIE GAMES SUMMIT

Page 17: Unreal conference slides

TEST FLOW DIAGRAMS

• Graphic model of a function or use case•Take the player’s point of view• Use loops• Add reverse flows• Can double as a design tool

INDIE GAMES SUMMIT

Page 18: Unreal conference slides

TEST FLOW DIAGRAMS

• Start with an entry point and initial “state”

IN

AMMOFULL

1. Start

INDIE GAMES SUMMIT

Page 19: Unreal conference slides

TEST FLOW DIAGRAMS

• Do something to change to another state

IN

AMMOFULL

NOAMMO

2. Fire All Ammo

1. Start

INDIE GAMES SUMMIT

Page 20: Unreal conference slides

TEST FLOW DIAGRAMS

• Can you go back to where you started?

IN

AMMOFULL

NOAMMO

2. Fire All Ammo

3.Reload

1. Start

INDIE GAMES SUMMIT

Page 21: Unreal conference slides

TEST FLOW DIAGRAMS

• Can you repeat anything?

IN

AMMOFULL

NOAMMO

4. Fire Weapon

5.Reload

2. Fire All Ammo

3.Reload

1. Start

INDIE GAMES SUMMIT

Page 22: Unreal conference slides

TEST FLOW DIAGRAMS

• Can you do something a different way?

IN

AMMOFULL

NOAMMO

4. Fire Weapon

5.Reload

2. Fire All Ammo

3.Reload

1. Start

6. Pick up fresh weapon

INDIE GAMES SUMMIT

Page 23: Unreal conference slides

TEST FLOW DIAGRAMS

• Provide an Exit point

IN

AMMOFULL

NOAMMO

4. Fire Weapon

5.Reload

2. Fire All Ammo

3.Reload

1. Start

6. Pick up fresh weapon OUT

7. Exit

INDIE GAMES SUMMIT

Page 24: Unreal conference slides

TEST FLOW DIAGRAMS

• Create tests from various paths

IN

AMMOFULL

NOAMMO

4. Fire Weapon

5.Reload

2. Fire All Ammo

3.Reload

1. Start

6. Pick up fresh weapon OUT

7. Exit

1,2,4,71,2,3,2,71,2,6,2,71,2,3,5,2,71,5,2,3,5,2,4,6,2,7

INDIE GAMES SUMMIT

Page 25: Unreal conference slides

ANY QUESTIONS?

Page 26: Unreal conference slides

EXTRAS

Page 27: Unreal conference slides

ESCAPED PROBLEMS

CONFIGURATION

Unreal Tournament 3 flickers after playing for some time / UT3 Flickering screenSolution A : Make sure that you are using the latest drivers.Solution B : Turn down / Turn-ff Anisotropic Filtering in UT3′s menu

Unreal Tournament 3 models are the same for all players after playing the first map.Make sure that the ‘Allow custom characters’ option is checked in the options. This is because there are times that the config dont get saved or some of the items in the options menu gets randomly checked. Make sure that the UT3 config is changed after ticking the “allow custom characters” option by checking the options menu again after closing.

Page 28: Unreal conference slides

CONFIGURATION

Unreal Tournament 3 Configs dont get saved / UT3 settings and cd-key not saved / cd-key problem Make sure that the MY DOCUMENTS or and UT3 folder is not set as read-only

Cannot Play Unreal Tournament 3 Campaign Open the UTEngine.ini (My Documents\My Games\Unreal Tournament 3\UTGame\Config) and change PumpWindowMessagesWhenRenderThreadStalled=False to PumpWindowMessagesWhenRenderThreadStalled=True

ESCAPED PROBLEMS