How to Break Software: Robustness Edition

TH AM Tutorial

10/1/2013 8:30:00 AM

"How to Break Software:

Robustness Edition"

Presented by:

Dawn Haynes

PerfTestPlus, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Dawn Haynes

PerfTestPlus, Inc.

Dawn Haynes is COO, principal trainer, and consultant for PerfTestPlus, Inc., and a former

director of the Association for Software Testing. Dawn’s unique blend of experience, humor, and

effectiveness at providing tools and techniques that help students at all levels generate new

approaches to common and complex software testing problems has resulted in her international

recognition as an elite trainer of testers. She provides consulting services and is a frequent

speaker at testing conferences, local groups, and intimate gatherings of testers.

1

© 2013 PerfTestPlus, Inc.


2



3



4



5



RandomRandom

StrikesStrikes

ExploratoryExploratory

TestingTesting

FaultFault

InjectionInjection

BugBug

SafariSafari

ErrorError

GuessingGuessing

6



7



8


Description MilesDescription Miles


9



10


© 2013 PerfTestPlus, Inc.Ref: How to Break Software - Whittaker

• Directed and focused attempt to evaluate the quality, especially reliability, of a test object by attempting to force specific failures to occur.

Attack

11


Input

� Force all error messages to occur

� Force software to use default values

� Explore data types & character sets

� Overflow input buffers

� Find interacting inputs

� Repeat inputs/sequence

Output

� Force different outputs to be generated for same input

� Force invalid outputs to be generated

� Force output properties to change

� Force the screen to be refreshed

Data

� Apply inputs using a variety of initial conditions

� Force a data structure to store too many or too few values

� Try to modify alternate data constraints

Computation

� Experiment with invalid operand and operator combinations

� Exploit recursion

� Force computation results to be too large or too small

� Find features that share data or interact poorly

Ref: How to Break Software - Whittaker


12



File

� Fill the file system to capacity

� Force media to be busy or unavailable

� Damage the media

� Assign invalid file name

� Vary access permissions

� Vary/corrupt file contents

Kernel � Restrict or constrain resources like memory, CPU, threading, etc.

Other

Software

� Make unavailable (uninstall, move or rename files, kill process)

� Restrict or block access (disable methods, close ports, etc.)

� Concurrency check (execute actions in other interacting software)

Ref: How to Break Software - Whittaker

13



• A test design technique where the experience of the tester is used to anticipate what defects might be present in the component or system under test as a result of errors made, and to design tests specifically to expose them.

Error Guessing

14



Business rules; logic

• Subvert rules

• Bend/break logic

Workflows; state models

• Skip steps

• Illegal U-turn

• Create zombies

• Interacting models

Concurrency• Doing the same thing

• Doing different things

Algorithms• Bust sorting logic

• Trick search operations

• Subvert parsing schemes

15



16



17



18



19



• Testing to determine the robustness of the software product.

Robustness Robustness TestingTesting

• The process of testing to determine the reliability of a software product.

Reliability Reliability TestingTesting

• An attribute of a component or system specified or implied by requirements documentation (for example reliability, usability or design constraints). [After IEEE 1008]

FeatureFeature

Source: ISTQB Glossary, 2007

20

© 2013 PerfTestPlus, Inc.Source: ISTQB Glossary, 2007

• Confirmation by examination and through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled. [ISO 9000]

ValidationValidation

• The capability of the software product to maintain a specified level of performance in cases of software faults (defects) or of infringement of its specified interface. [ISO 9126] See also reliability, robustness.

Fault Fault ToleranceTolerance


21



22



23



24



25

© 2013 PerfTestPlus, Inc.Source: ISTQB Glossary, 2007


The ability of software to The ability of software to deliver value during normal deliver value during normal use without stressing out use without stressing out

users (or failing)users (or failing)

Robustness

The ability of software to The ability of software to The ability of software to The ability of software to deliver value consistently deliver value consistently ––throughout the longevity, throughout the longevity,

intermittence, or frequency of intermittence, or frequency of interactionsinteractions

Reliability

Technology

How to Break Software: Robustness Edition