The Craft of Bug Investigation

TD AM Tutorial

10/1/2013 8:30:00 AM

"The Craft of Bug Investigation"

Presented by:

Jon Bach

eBay

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Jon Bach

eBay, Inc.

With more than eighteen years of experience in software testing, Jon Bach has held technical

and managerial positions in companies including Hewlett-Packard and Microsoft. In his current

role as director of Live Site Quality for eBay, Jon is dedicated to building “end-to-end” tests

(activity flows) in eBay’s core sites to discover important bugs that threaten its core business.

He is most notable for creating, with his brother James, Session-Based Test Management, a

method to manage and report exploratory testing.

1

The Craft of (black box)

Bug Investigation

Jon Bach

QE Director, eBay

[email protected]

STAR West 2013

eBay Inc. confidential

Preamble

These next few hours are designed with

exercises that lead you through ways of

thinking that help you narrow down a bug.

We’ll discuss techniques for bug isolation,

reproducing intermittent bugs, and skills to

help you know when you’ve gone as far as

is reasonable before handing it off.

mailto:[email protected]

2


A morning of investigation

Psychic Reader

Checkout Checking Out?

Art Show

Mysterious Spheres

Out-of-Bounds

Gettysbug Address


CIDTESTDSFDPOTCRUSSPICSTMPLFDSFSCURA

Product Elements

Structure

Function

Data

Platform

Operations

Time

General Test Techniques

Function testing

Domain testing

Stress testing

Flow testing

Scenario testing

Claims testing

User testing

Risk testing

Automatic testing

Project Environment

Customers

Information

Developer relations

Team

Equipment & tools

Schedule

Test Items

Deliverables

Quality Criteria

Capability

Reliability

Usability

Security

Scalability

Performance

Installability

Compatibility

Ways we find bugs

} } } } }

Development Criteria

Supportability

Testability

Maintainability

Portability

Localizability

3


How *well* do we find bugs?

Some Investigation Skills and Tactics

Modeling

Resourcing

Questioning

Recording

Reporting

Investigation is a mindset using this skillset.

Chartering

Observing

Manipulating

Collaboration

Generating/Elaborating

Overproduction/Abandonment

Abandonment/Recovery

Refocusing

Alternating

Branching/Backtracking

Conjecturing

“MR.Q COMC GOARABC R&R?”


How well do we investigate them?

Good enough quality bar

Further time and effort

is crucial.


is a waste of resources.

Unacceptable Quality

Too Much Quality

floating line

4


Abductive Inference

1. Collect data.

2. Find several explanations that account for the data.

3. Find more data that is either consistent or inconsistent with explanations.

4. Choose the best explanation that accounts for the important data, or keep searching.

Abductive inference means finding the best

explanation for a set of data.

Jump to conjectures, not conclusions

eBay Inc. confidential 8

This is a dip in bids we saw in Production.

What questions do you have to troubleshoot this?

Is Checkout Checking Out?

5

eBay Inc. confidential 9

Topics for investigation (FORCOSTTTIP)

• Feature – Do we know what happens when people bid?

• Origin -- When did this start happening?

• Recent – Did we just push code to Production?

• Consistency / Correlation -- What happened yesterday, tomorrow? Are there

other graphs and metrics?

• Overlap – Is there any other process happening?

• Security – Is it malicious? Intentional?

• Terminology -- What kinds of “bids”? US, UK, International?

• Third-Party / Silent Partner – What’s not in the room? What sort of information

have we NOT heard from?

• Truth is out there – does someone already know about this?

• Instrumentation – is the graph wrong? Can I get more data?

• Prediction – can we test a conjecture with a prediction


Exercise: Psychic Reader

../STP 2009/CSI/psychic.swf

6


Lessons

You can be psychic, too

You may be clue-less, for now

Try “defocusing”

Are you just following orders?


Exercise: Mysterious Spheres

7


Lessons

If it fails, whose fault is it?

Assumptions need nourishment

Oracles are in “control”

“Elsewhere”, is a useful thought


Exercise: Out-of-Bounds

../Ideas & Intellect/Exercises/IP/Happy-sad with Mike Butler.xls

../Ideas & Intellect/Exercises/tri_new/tri-bach2.exe

8


Lessons

Be a “model” tester

Stagehands work behind curtains

Outside data is not inside data

Computers are our slaves


Exercise: Art Show

What is my Operating Rule?

What do you notice?

What (crazy) test haven’t you tried?

9


Lessons

“You’re so one dimensional”

Jump to *conjectures*

Yes, I *do* look FAT in this. (M vs. O)

Breaking “rules”!


Exercise: Gettysbug Address

Does it work?

What is the hidden feature?

What story does the data tell?

../Ideas & Intellect/Exercises/IP/IP_Exercise_results.xls

../Ideas & Intellect/Exercises/IP/ip_address.exe

10


Lessons

The Rumble Strip heuristic

Missing a mission?

Be “thoughtless” (random is good)

Defocus: do *anything*, *differently*


Exercise: What’s the Pattern?

Investigation of behavior is like

trying to repro a bug… what is

the behavior of this app?

../Ideas & Intellect/Exercises/pattern/pattern.exe

11


Lessons

Checking is different than testing

Collaboration is a kick

Not all characters are created equal

Description takes practice


How well do we investigate them?

Good enough quality bar


is crucial.


is a waste of resources.

Unacceptable Quality

Too Much Quality

floating line

12


How much investigation?

The answer must be “Yes” to all four criteria,

but ask “for whom, what, and when?”

1) Sufficient benefits

2) No critical problems

3) The benefits outweigh the problems

4) In the present situation, and all things

considered, improvement would be more harmful

than helpful


Intermittent Bugs: Observation

Link

Bad observation

Irrelevant observation

Bad memory

Misattribution

Misrepresentation

Unreliable oracle

Unreliable communication

C:/Documents and Settings/jonb/Desktop/alaska class/Investigate Intermittent Problems.htm

13


Intermittent Bugs: System

Purposeful change, and then back to original

Accidental change

Platform change

Flaky hardware

Trespassing system

Executable corruption

Component competition


Intermittent Bugs: Machine State

Frozen conditional

Improper Initialization

Resource denial

Progressive data corruption

Progressive destabilization

Overflow

Occasional functions

Different mode or option setting

14


Intermittent Bugs: Input

Accidental

Secret boundaries or conditions

Different profile

Ghost input (alternative source)

Simultaneous action as others

Compromised input

Timing issues

Crazy Combinations


Intermittent Bugs: You?

You may not be aware of...

…variables of influence

…sources of distortion in your observations

…available tools that might help

…boundaries and their characteristics

…the system's missing / extra functions

…complex / competing algorithms

15


A few (free) investigation tools

Httpwatch

-- displays log of requests and responses

Firebug

– inspect HTML

-- JavaScript debugger

WAVE – accessibility errors


More info

• ET Dynamics:

http://www.satisfice.com/articles/et-dynamics.pdf

• Test Heuristics and Planning

http://www.satisfice.com

• Context-Driven Software Testing

http://groups.yahoo.com/group/software-testing

• Center for Software Testing Education and Research

http://www.testingeducation.org/BBST

• Books related to Exploratory Testing skills and tactics

http://www.testingreflections.com/node/view/3190

• Scenario testing examples

– http://www.testingeducation.org/a/scenario2.pdf







http://www.testingeducation.org/BBST

http://www.testingreflections.com/node/view/3190

Technology

The Craft of Bug Investigation