nt Session 

 

Presented by: 

Gerie Owen, Northeast Utilities, Inc 

  

Brought to you by: 

  

340 Corporate Way, Suite   Orange Park, FL 32073 888‐2

T12 Concurre4/8/2014   2:00 PM     

“How Did I Miss That Bug?  Managing Cognitive Bias in Testing” 

 

 

Peter Varhol, Telerik   

    

300,68‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com 

Gerie Owen Northeast Utilities, Inc  

QA Consultant Gerie Owen specializes in developing and managing offshore test teams. Gerie has implemented the offshore model—developing, training, and mentoring new teams from their inception. She manages large, complex projects involving multiple applications; coordinates test teams across multiple time zones; and delivers high quality products. Gerie’s most successful project team wrote, automated, and executed more than 80,000 test cases for two suites of web applications, allowing only one defect to escape into production. In her everyday life, Gerie enjoys mentoring new test leads and brings a cohesive team approach to testing.

Peter Varhol Telerik  

A testing evangelist at Telerik, Peter Varhol is a writer and speaker on software development and quality topics. Peter has authored dozens of articles on software tools and techniques for building applications, and has given conference presentations and webcasts on a variety of topics—user-centered design, integrating testing into agile development, and building the right software in an era of changing requirements. He has held key roles on engineering teams that have produced award-winning, quality tools such as BoundsChecker and SoftICE. Peter’s past roles include technology journalist, software product manager, software developer, and university professor.

2/4/2014

1

How Did I Miss That Bug?How Did I Miss That Bug?

OOOvercome Cognitive Bias Overcome Cognitive Bias In TestingIn Testing

Peter VarholPeter.@petervarhol.com

Gerie OwenGerie.owen@gerieowen.com

Why are we talking about Why are we talking about missing bugs?missing bugs?g gg g

• Have you ever missed a bug?• Have you ever been asked how you missed a bug?• Have you ever wondered how you missed a bug?

2/4/2014

2

Consequences of Missed Consequences of Missed BugsBugs

• Possible Consequences of Missed Bugs:o Negative Publicityo Lost Saleso Lost Customerso Even Loss of Life

MISSED BUGS CAUSE MAYHEM

My JourneyMy JourneyThe “HOW” is more important than the “WHY”The HOW is more important than the WHY

And now , I invite you to join with me into the journey of

How Did I Miss that Bug?How Did I Miss that Bug?

2/4/2014

3

How Do We Miss Bugs?How Do We Miss Bugs?• Missed test casesMissed test cases• Misunderstanding of requirements• Misjudgment in risk-based testing• Inattention• Fatigue• Burnout• Multi-tasking

How Do We Test?How Do We Test?• What is Software Testing?g

o Software testing is making judgments about the quality of the software under test.

o Involves:• Objective comparisons of code to

specifications,• AND• Subjective assessments regarding usability,

functionality etc

2/4/2014

4

What IS What IS aa Missed Bug?Missed Bug?

An Error in Judgment!To determine how testers miss bugs, we need to understand how humans make judgments, especially in complex situationsin complex situations.

2/4/2014

5

How do we make How do we make judgments?judgments?

• Thinking, Fast and Slow – Daniel Kahneman

o System 1 thinking – fast, intuitive, and sometimes wrong

o System 2 thinking – slower, more deliberate, more accuratemore accurate

System 1 vs. System 2 System 1 vs. System 2 ThinkingThinking

• System 1 thinking keeps us functioningo Fast decisions, usually right enougho Gullible and biased

• System 2 makes deliberate, thoughtful decisionso It is in charge of doubt and o It is in charge of doubt and

unbelievingo But is often lazyo Difficult to engage

2/4/2014

6

How Do We Apply System 1 How Do We Apply System 1 and System 2 Thinking?and System 2 Thinking?

• System 1 thinking:• System 1 thinking:o Is applied in our initial reactions to situations. o May employ Heuristics or rules of thumb

• System 2 thinking:o Is applied when we analyze a problem, for example

when calculating the answer to a math problem.

S t 1 d S t 2 b i fli t• System 1 and System 2 can be in conflict:o lead to biases in decision-making.

The Lily Pad QuestionThe Lily Pad Question

In a lake, there is a patch of lily pads. Every day, the patch doubles in size.

If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover

f fhalf of the lake?

2/4/2014

7

How do Biases Impact How do Biases Impact Testing?Testing?

• We maintain certain beliefs in testing practice• We maintain certain beliefs in testing practiceo Which may or may not be factually trueo Those biases can affect our testing resultso We may be predisposed to believe something that

affects our work and our conclusions

Ho do bias and error ork?• How do bias and error work?o We may test the wrong thingso Not find errors, or find false errors

The Representative BiasThe Representative Bias• Happens when we judge the likelihood of an Happens when we judge the likelihood of an

occurrence in a particular situation by how closely the situation resembles similar situations.

• Testers may be influenced by this bias when designing data matrices, perhaps not testing data in all states or not testing enough types of data in all states or not testing enough types of data.

• Case Study: Completed Order bug

2/4/2014

8

The Curse of KnowledgeThe Curse of Knowledge• Happens when we are so knowledgeable about pp g

something, that our ability to address it from a less informed, more neutral perspective is diminished.

• When testers develop so much domain knowledge that they fail to test from the perspective of a new user. Usability bugs are often missed due to this user. Usability bugs are often missed due to this bias.

• Case Study: Date of Death Bug

The Congruence The Congruence BBiasias• The tendency of experimenters to plan and execute The tendency of experimenters to plan and execute

tests on just their own hypotheses without considering alternative hypotheses.

• This bias is often the root cause of missed negative test cases. Testers write test cases to validate that the functionality works according to the

ifi ti d l t t lid t th t th specifications and neglect to validate that the functionality doesn’t work in ways that it should not.

• Case Study: Your negative test case or boundary miss

2/4/2014

9

The Confirmation BiasThe Confirmation Bias• The tendency to search for and interpret The tendency to search for and interpret

information in a way that confirms one’s initial perceptions.

• Testers’ initial perceptions of the quality of code, the quality of the requirements and the capabilities of developers can impact the ways in which they test developers can impact the ways in which they test.

• Case Study: Ability to print more than once bug

The Anchoring The Anchoring EEffectffect• The tendency to become locked on and rely too y y

heavily on one piece of information and therefore exclude other ideas or evidence that contradicts the initial information.

• Software testers do this often when they validate code to specifications exclusively without code to specifications exclusively without considering ambiguities or errors in the requirements.

2/4/2014

10

Inattentional BlindnessInattentional Blindness• Chabris and Simon conducted experiments on how

focusing on one thing makes us blind to othersfocusing on one thing makes us blind to otherso Invisible gorilla on the basketball courto Images on a lung x-ray

Inattentional BlindnessInattentional Blindness• a psychological lack of attention

• the tendency to miss obvious inconsistencies when focusing specifically on a particular task.

• This happens in software testing when testers miss the blatantly obvious bugs

2/4/2014

11

WWhy Do We hy Do We Develop Develop BiasesBiases??

• The Blind Spot Biaso We evaluate our own decision-making process

differently than we evaluate how others make decisions.

• West Meserve and StanovichWest, Meserve and Stanovich

Are you subject to Bias?Are you subject to Bias?A Test Planning Exercise

This application is used to send a series of letters to This application is used to send a series of letters to homeowners and renters to inform them of required inspections and get them to book appointments. The series are different based on whether the residence is a single or multi family and ends with a certified letter.

The bug is that if the renter to whom the initial letter was sent moved the new renter would get the next letter in a sent moved, the new renter would get the next letter in a series, potentially a strongly-worded certified letter when the series should start again from the first letter.

This is a medium-severity defect; but high priority as it negatively affects customer experience.

2/4/2014

12

How Does This Apply To How Does This Apply To Testing?Testing?

W t th • We must manage the way we think throughout the test process.

oAs individual testersoAs individual testersoAs test managersoAs a professional community

How Can Testers Manage How Can Testers Manage Their Thought Processes?Their Thought Processes?• Use more System 1 thinking?

OR• Use more System 2 thinking?

2/4/2014

13

Test Methodology and Test Methodology and System 2 ThinkingSystem 2 Thinking

• Test methodology is the analytical Test methodology is the analytical framework of testing; it invokes our system 2 thinking and places the tester under cognitive load.

Th d t i ti f h th th t l • The determination of whether the actual results match the expected results becomes an objective assessment.

HHow ow DDo o WWe e FFind ind BBugs?ugs?

Focus on System 1 thinking Focus on System 1 thinking, intuition and emotion

2/4/2014

14

Focus On System 1 Focus On System 1 ThinkingThinking

He ristics sed ith Oracles• Heuristics used with Oracles• Recognize our emotions as indicators

of potential bugs• Exploratory Testing

How Should We Use How Should We Use Exploratory Testing?Exploratory Testing?

• Unstructuredo Before beginning test case execution

• Minimizes preconceived notions about the application under test

o Oracle basedo Oracle based• Users’ perspectives• Data flow

2/4/2014

15

How Should We Use How Should We Use Exploratory Testing?Exploratory Testing?

• Structuredo Use to create additional test cases

• May be done earlier, possible as modules are developed

o Session-Based• Time-boxed charters• Multiple testers • Post test review session

What Can Test Managers What Can Test Managers Do?Do?

• Foster an environment in which the testers feel comfortable and empowered to use System 1 thinking.

o Plan for exploratory testing in the test scheduleo Encourage Testers to take riskso Reward for Quality of bugs rather than quantity

of test cases executed

2/4/2014

16

What Can the QA Profession Do?What Can the QA Profession Do?A Paradigm Shift A Paradigm Shift 

o Shift our focus from requirements coverage based test execution to a more intuitive approach

o Exploratory testing and business process flow testing becomes the norm rather than the exceptionexception

o Develop new testing frameworks where risk-based testing is executed through targeted exploratory testing and is balanced with scripted testing

Question Test ResultsQuestion Test Results• Is there any reason to suspect we are evaluating

our test results based on self-interest, overconfidence, or attachment to past experiences?

• Have we fallen in love with our test results?• Were there any differences of opinion among the

team reviewing the test results?

2/4/2014

17

How Do We Find The How Do We Find The Obvious Bugs?Obvious Bugs?

• Focus less• Use intuition• Believe what we can’t believe