Chapter 22Developer testing

Peter J. Lane

Testing can be difficult for developers to follow

Testing’s goal runs counter to the goals of the other development activities. The goal in testing a software is to break it.

It can never completely prove the absence of errors.

Testing alone does not improve the software’s quality. Fix it after you test it.

You are required to assume that you’ll find errors in your code.

If you assume you won’t, then that is more than likely so. Even when there are errors.

Testing During Construction“White/glass – box” testing

Can look into the class while testing

“Black – Box” testing See no further than the interface while testing

Easier to debug if you test a routine or class one at a time vs. several at once.

Recommended approach, cover this ground:Test each relevant requirement to make sure

that the requirements have been implemented.

Test each relevant design concern to make sure that the design has been implemented.

“Basis testing” adds detailed test cases to those that the test requirements and the design.

Make a checklist for past errors.

Test First or Last?Writing test cases before code does not take any more effort

than after.

If first, you can detect defects earlier and you can correct them more easily.

Writing cases first forces you to think a little bit about the requirements and design.

It exposes requirement problems sooner before code is written.

LimitationsDeveloper tests tend to be “clean” tests.

Tries to how it works rather than how it breaks.

Tends to have an optimistic view of test coverage. Most programmers assume they’ve covered more

than they actually did,

Tends to skip more sophisticated kinds of test coverage.

Incomplete TestingPick test cases that are most likely to find

errors.

Concentrate on picking a few that tell you different things instead of a set that says the same thing again and again.

Structured Basis Covering Start with 1 test case for the routine itself.

Add 1 test case for every logic statement (if, while, for, and and or)

Data-Flow TestingDefined – The data has been initialized, but

not used yet.

Used – Data that’s been used for computation.

Killed – Once defined data that has now been undefined.

Entered – Control flow enters routine immediately before the variable is acted upon.

Exited – The control flow leaves the routine immediately after the variable acted upon.

Equivalence Partitioning If two test cases bring out exactly the same

errors, you need only one of them.

Helps when you are not looking at it from source code.

Error GuessingHeuristic approach

Sophisticated guessing.

Boundary analysis & Compound BoundariesTest cases that exercise the maximum and

minimum of allowable values.

Classes of Bad Data

Too little data (or no data)

Too much data

Invalid data

Wrong size of data

Uninitialized data

Classes of Good DataNominal cases – middle-of-the-road, expected

values

Min and Max normal configurations

Compatibility with old data

Test Cases that make Hand checks convenientUsing numbers in values that are easy to

calculate by hand for testing.

Which classes contain most errors80%of the errors are found in 20% of a

project’s classes or routines.

50% of the errors are found in 5% of a project’s classes.

Errors by ClassificationScope of most errors is fairly limited

Many errors are outside the domain of construction Thin application domain knowledge, conflicting requirements, bad

communication

Most construction errors are the programmer’s fault 95% programmer, 2% system software, 2% other software, 1%

hardware

Typos are surprisingly common source problems

Misunderstanding the design is a recurring theme in studies of programmers errors

Most are easy to fix

It’s a good idea to measure your own organization’s experience with errors.

Proportion from Faulty ConstructionOn small projects, construction defects make

up the vast bulk of all errors.

Construction defects account for at least 35% of all projects regardless of size.

Although cheaper to fix than requirements and design errors, construction errors are still expensive.

Errors in Testing ItselfCheck your work

Plan test cases as you develop your software

Keep your test cases

Plug unit tests into a test framework

Chapter 23Debugging

The scientific Method of Debugging1. Stabilize the error

Unpredictable errors are often initialization errors Requires narrowing a test case that produces the error.

2. Locate the source of error Gather data that produces defect Analyze the data that has been gathered, and form a

hypothesis about the defect Determine how to prove/disprove the hypothesis, by testing or

examining the code and do so.

3. Fix defect

4. Test fix

5. Look for similar errors

Tips for finding defectsUse all the data available to make hypothesis

Refine the test case that produce the error

Exercise the code in your unit test suite Test code in isolation

Use available tools

Reproduce the error several different ways

Tips (part 2)Generate more data to generate more

hypotheses

Use results of negative tests

Brainstorm for possible hypotheses

Make list of things to try

Narrow the suspicious region of the code

Tips (part 3)Be suspicious of classes and routines that have had

defects before

Check code that’s changed recently

Expand the suspicious region of the code

Integrate incrementally

Check for common defects

Talk to someone else about the problem

Take a break from the problem

Syntax ErrorsDon’t trust line numbers in compiler messages

Compiler may place error on the line before or after the actual error.

Don’t trust compiler messages May need to read between the lines to understand

Don’t trust compiler’s second message Errors following the first may no mean anything

Divide and conquer Temporarily remove part of code and compile again.

Find misplaced comments and quotation marks

Fixing defects

Understand the problem before you fix it

Understand the program, not just the problem

Confirm the defect diagnosis

Relax

Save original source code

Fix the problem, not the symptom

Change code only for good reasons; be certain that it will work

One change at a time

….. Okay, I’m done.