Alan Page: On Testing

TF AM Tutorial

10/1/2013 8:30:00 AM

"Alan Page: On Testing"

Presented by:

Alan Page

Microsoft Corporation

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Alan Page

Microsoft

Alan Page is a principal SDET—a fancy name for tester—on the Xbox console team at

Microsoft. Edging up on twenty years in software testing, Alan has previously worked on a

variety of Microsoft products including Windows, Windows CE, Internet Explorer, and Office

Lync. He spent some time as Microsoft’s director of test excellence where he developed and ran

technical training programs for testers throughout the company.

8/27/2013

1

Alan Page: On Testing

Test Ideas are…Ideas

8/27/2013

2

8/27/2013

3

The Adjacent Possible

Liquid Networks

The Slow Hunch

Serendipity

Error

Exaptation

Platforms

On Test Automation

On Exploratory Testing

On Test DesignOn Test Careers

On the Value of Test

On Test Management

On working with others

On The Death of Test

On the Future of Test

On Coding Testers

On Test Communities

On Agile and Testing

On Writing On Learning On Leading On Innovation

8/27/2013

4

Test Automation

8/27/2013

5

Exercise 1

8/27/2013

6

Exploratory Testing

8/27/2013

7

See Test DesignSee Test Design

Test Design

8/27/2013

8

Test This

What is Test Design?Testing Ideas

8/27/2013

9

Critique Product

Support Programming

B us iness (Customer ) Fac ing

Techno l og y Fac ing

Testing Quadrants: Marick, Crispin, Gregory, Hendrickson, others…

Runtime Analysis Tools

Functional Testing

Acceptance Tests

Scenarios

Usability Testing

World Readiness

Exploratory Testing

Take home / Beta Testing

Unit Testing

Code Coverage

Static Analysis

Code Churn Analysis

Performance Testing

Security / Privacy Testing

Stress Testing

Critique Product

Support Programming

B us iness (Customer ) Fac ing

Techno l og y Fac ing

Testing Quadrants: Marick, Crispin, Gregory, Hendrickson, others…

8/27/2013

10

Functionality First?

Why are we looking for functional bugs? They're easy. Stop looking for functional bugs. @docjamesw

“…testing, for a long time and in many places, has been myopically focused on functional correctness, rather than on value to people.”

-Michael Bolton

Random Internet Quotes “I don't care about features or plug-ins or add-ons or whatever. Just load the page quickly and let me get on with my work”

“ <some company> makes wonderfully full featured and functionally correct software, but my wife can’t figure out how to use it”

8/27/2013

11

Customer Focused Test Design

Customer Focused Test Design

8/27/2013

12

Upside Down Testing

8/27/2013

13

The Cost of Bugs

Is The Cost Curve Always True?

8/27/2013

14

Hypothetical Cost Curve –Functional Bugs

Hypothetical Cost Curve –Design Bugs

8/27/2013

15

Security Privacy

Reliability Performance

Usability Globalization

Reliabi l i ty

Send Web Query 10,000 times

Send malformed web query 10,000 time

8/27/2013

16

P e r f o rm a n c e

Test timing / latency of an action or scenario

Test large number of simultaneous connections

Scenario Testing

Customer Experience Validation

8/27/2013

17

Customer Experience Validation

Assess the experience

Collaborate with user research

Balance with other approaches

Evaluating a ScenarioEase of use

Responsiveness

Usefullness

Visual Appeal

8/27/2013

18

Example:

Showcase (marketing speak)

Scenario (project manager speak)

End-to-End Test (something

actionable)

(Optional) Variations

Xbox Knows You Better Identity

Step in front of the console, get recognized and see

my curated content

Light/Dark settings

Child/Adult/Gender/Height/Size/Apparel

Curated content

Ease of Use

• Am I able to complete the scenario?

• Is it complicated? Do I need to perform extraordinary steps to get what I need done?

• Are there glitches in the system that make it difficult?

• Was it hard to find how to execute this scenario? Are the features hidden?

• Is the experience consistent?

8/27/2013

19

Responsiveness

• Did I feel like the scenario was fast and fluid?

• At any point did I feel like I had to wait a frustrating amount of time?

• Was I effectively distracted while waiting for an action to take place? (ex. Movie or animation while I wait)

Usefulness

• Would you use this scenario yourself to accomplish this specific outcome?

• Does this scenario meet a need for our consumer?

• Does this let me do something that I want to do?

8/27/2013

20

Visual Appeal

• Is the experience exciting to see and hear?

• Is the UI polished?

• Does the UX make the experience enjoyable?

Evaluation Scale5 Love It! – You love it so much you’d shout it out at the top of

your lungs through a bullhorn from every rooftop you encounter

4 Like It – You like it a lot and might mention it during a lull in conversation at a dinner party

3 Meh – You can live with it and neither like it or hate it. It’s

nothing special. There are some improvements that can be made

2 Don’t Like It – The experience leaves a bad taste in your

mouth. You’d use it if you really had to otherwise, you’d stay away from it.

1 Hate it! – You hate it so much that you would only use it if you were under a Hogwarts compulsion spell

0 Not Implemented

8/27/2013

21

Scenario Love It Like It Meh Don't Like Hate

Do this 43 10 1 1 1

Do that 3 2 9 14 20

Do the other thing 40 14 5 0 1

Do something else 20 3 5 7 15

Test Management

8/27/2013

22

What is the role of a manager?What is the role of a manager?

What are the activities of a manager?What are the activities of a manager?

learn your A-C-B’s

8/27/2013

23

find the steepest learning curve

ride the gravy train

8/27/2013

24

who do you know?

follow the leader - lead the follower

8/27/2013

25

find a mentor

never let yourself get blocked

8/27/2013

26

try a new way

don‘t flip the bozo bit

8/27/2013

27

…and don’t burn bridges either

the three p’s

8/27/2013

28

a career is a journey, not a sprint

have fun