Fixing the DefectCEN4072 – Software Testing

Yuriy Zaynulin, 2015 2

From Defect to Failure

How a defect becomes a failure:

1. The programmer creates a defect

2. The defect causes an infection

3. The infection propagates

4. The infection causes a failure

Yuriy Zaynulin, 2015 4

From Defect to Failure (cont.)

How to locate a defect:

1. Identify infected value that defines the failure (pic. [a])

2. Determine the possible origins of the infected value (pic. [b])

3. Check every possible origin for infection (pic. [c])

Found the earlier infection? Go to step 2To

picture

Yuriy Zaynulin, 2015 5

Techniques for Locating Defects

Infections e.g. a failed assertion

Causes e.g. “a[2] = 0” causes the failure

Anomalies e.g. f() executed only in failing run

Code Smells e.g. uninitialized variable

Dependencies e.g. a[2] comes from a[0]

Yuriy Zaynulin, 2015 6

Techniques for Locating Defects (cont.)

Assertions: ensures data sanity

failing assertion signals an infection

passing assertion needs not be considered

Anomalies - aspects of code correlated with failure

Causes - aspects of execution correlated with failure AND causing the

failure

Yuriy Zaynulin, 2015 7

Techniques: Dependences

Dependences: check the backward slice for infections

start with the closest statements

anything not in the backward slice

CANNOT have caused the infection

[x] is a dependency of [!]s

Yuriy Zaynulin, 2015 8

Techniques: Observation

Observation: allows a programmer to tell (or specify)

what is correct and what isn’t correct

Observe:transition from SANE to

INFECTED

Yuriy Zaynulin, 2015 9

Techniques: Ordering

1.Focus on Infections [ if an origin is already known to be faulty, else: ]

2.Focus on Causes [ if delta debugging or experimentation found the cause of a failure, else: ]

3.Focus on Anomalies [ if they are present, else: ]

4.Focus on Code Smells [ if they are present, else: ]

5.Focus on Dependences [ if nothing else worked, examine backward slices; ]

Yuriy Zaynulin, 2015 10

Is the Error a Cause?

Origin is an error, but doesn’t cause the failure?

→ Wrong track!

Example:

a = compute_value();printf("a = %d\n", a);

output is 0

we assume compute_value() produces infection

set a to 1 right before printf(...), still output is 0

→ actual problem is with printf(...)

Yuriy Zaynulin, 2015 12

Is the Cause an Error?

Origin causes the failure, but not [sure whether it is] an error?

→ DO NOT debug it into existence!

Example:To example

Yuriy Zaynulin, 2015 13

Devil’s Guide to Debugging

1. Find the defect by guessing:

Scatter debug statements all over

Try changing code until something works

Don’t backup

2. Don’t waste time understanding the problem

3. Use the most obvious fix

Example:

x = compute(y); // compute() doesn’t work for y == 17, so fix it if (y == 17) x = 25.15;

Yuriy Zaynulin, 2015 14

Correcting the Defect

BEFORE: Save the code to Version Control System (VCS)

* BRIEF PERIOD OF HAPPINESS *

AFTER: Does the failure no longer occur?

Did the correction introduce new problems?

Was the same mistake made elsewhere?

Did I commit the change?

Yuriy Zaynulin, 2015 15

Workaround

Not a permanent solution

Defect DOES NOT cause a failure anymore

Defect DOES still remain

→ Keep the problem open

[in the tracing system]

Yuriy Zaynulin, 2015 16

Workaround (cont.)

When it is appropriate to use:

Unable to change

Supplied by third party

Closed source code

Risks

Large changes throughout the system needed

Flaw

Code is fine, but:

Design must undergo major revision

Yuriy Zaynulin, 2015 17

Workaround (cont.)

Famous workarounds: Spam filters

Flaw: anyone can easily forge email sender identity

Proper solution: redesign entire e-mail system [costs and risks]

Virus scanners

Flaw: OS users have administrator rights, any program can gain full control

Proper solution: assign users limited rights and add proper authorization [many good programs assume administrator right for installation]

Yuriy Zaynulin, 2015 18

Learning From Mistakes

Improve test suite

Tests failed to detect the problem → extend the tests.

Set up assertions

Keep assertions used in debugging. Add more. Leave assertions active in production code.

Improve training

Teach programmers about potential pitfalls and how to avoid them.

Improve the software process

Software was not tested fully? Wrong version was shipped? Critical part was not reviewed? Work on production process!

Improve analysis tools

Yuriy Zaynulin, 2015 19

Facts on Fixes

1. In the ECLIPSE and MOZILLA projects, about 30 to 40% of all changes are fixes.

2. Fixes are typically two to three times smaller than other changes.

3. Fixes are more likely to induce failures than other changes.

4. Only 4% of one-line changes introduce new errors in the code.

5. A module that is one year older than another module has 30% fewer errors.

6. Newly written code is 2.5 times as defect prone as old code.

Yuriy Zaynulin, 2015 20

Summary

To isolate infection: Work backward along infections origins

To find most likely origin: Focus on

Failing assertions

Causes

Anomalies

Code smells

Each origin must be an infection as well as a cause;

Use workaround if correction is too costly or risky

Yuriy Zaynulin, 2015 21

Summary (cont.)

To correct the defect: know how

Your change breaks the infection chain

Will the failure no longer occur

To ensure your correction is successful: check

The failure no longer occurs

No new problems were introduced

Yuriy Zaynulin, 2015 22

Sources

CADD Manager, “When Workarounds Become Best Practices”,http://www.caddmanager.com/CMB/2010/09/when-workarounds-become-best-practices/, 2010

Graves et al., 2000

Mockus, Votta, 2000

Mockus, Weiss, 2000

Ostrand, Weyuker, 2002

Purushothaman, Perry, 2004

Sliwerski et al., 2005

Zeller A., “Why Programs Fail: A Guide to Systematic Debugging”, 2005