March 30, 2004

Exploratory Testing

Testing Approaches Analytical Information-driven Intuitive Exploratory

Design the tests and test concurrently Learn the system and test it as you go Structure creative testing Think while testing Risk-based Analogous to Extreme Programming

Exploratory Testing Tasks Explore

Elements of the product How the product should work

Design Tests Which elements Speculate on possible quality problems

Execute Tests Observe behavior Evaluate against expectations

All with test design techniques best suited for the product

Exploratory Testing Practice Used to probe for weak areas Especially useful when

Weak specifications and requirements Little domain knowledge Time pressures

Less appropriate when Well-defined test requirements Strong need for regression testing Repeatable over releases

Cost of maintenance Few new test cases

Planning Decompose the product into elements

Areas of function that you can test in 1-2 days Define charters

Decomposition into units that can be tested in 1-2 hours

Quality criteria Capability, reliability, usability, performance,

installability, compatibility, …

Select test techniques

Charter

Provides clear mission of why this test Suggests what and how it should be

tested, as well as problems to look for Not a detailed plan, but should be as

specific as possible Might include risks, documents and

desired output

Maintenance

Cost of Maintenance

Estimates of percentage of total life cycle cost: 40% - 90%

Software Maintenance Types Adaptive maintenance: changes needed

as a consequence of operation system, hardware, or DBMS changes

Corrective maintenance: the identification and removal of faults in the software

Perfective maintenance: changes required as a result of user requests

Preventive maintenance: changes made to software to make it more maintainable

Why adaptation?

Lehman’s Law: if a program doesn’t adapt, it becomes increasingly useless Example: programs that didn’t adapt

to the web The majority of maintenance is

concerned with evolution deriving from user requested changes.

Lehman’s Second Law As an evolving program changes, its

structure tends to become more complex. Extra resources must be devoted to preserving

the semantic and simplifying the structure.

For most software, nothing has been done about it, so changes are increasingly more expensive and difficult.

How is Maintenance Initiated?

Most common: user request Types of requests

Bug reports Accepted Working as designed Documentation fix

Feature requests New environments

Steps for handling a change

Understand the problem Design the changes Analyze impact Implement changes Update documentation Regression test Release

Patches What is a patch?

Quick fix that doesn’t go through the full process

When should it be used? Error that is preventing use of the system

Problems with use Multiple patches can be order dependent Users can barely track which ones have been

applied Code version explosion Permanent fix may or may not be compatible

Problems of Maintenance Organizational

Alignment with objectives Cost benefit analysis

Process Impact Documentation Regression testing

Technical Building software that is maintainable

Different Objectives over Time

At release: bug-free Six months later: competitive or

competition-leading features Two years later: reduce

maintenance cost

Outsourcing of Maintenance Outsourcing: inside or outside the

company

Advantages Less expensive Uses less skilled people Frees developers for the next release

Disadvantages Changes can be slower May still require developer help Code can become less maintainable

Cost Benefit Analysis(Risk Analysis)

Will this problem reduce the number of programs that I sell?

Will this problem impact future sales/ How many people will it affect? How important are the customers it

will affect? Is it a “show stopper” or an

annoyance?

Process Issues

Impact analysis Updating documentation Reengineering

Legacy systems Reverse engineering

Impact analysis

How much of the system will it impact?

How complex are the changes? Heuristics:

Percentage of requirements that are impacted

Where was the defect injected (requirements, design, code)

Updating documentation A requirement defect can impact

Requirements Use cases Architecture Design Code

Where in the process do you update? No matter when, there is a period of

inconsistent documents!

Reengineering Code gets messy over time

Extreme programming re-factoring At some point, quality suffers

Changes slow Fixes introducing errors

Need to invest in the code! Rules as to when to rewrite a module Abstractions: variables -> methods Harder: when is REDESIGN needed?

Legacy Systems

Existing systems that are still useful May not want to invest in

enhancements Future functions will use new process

May not be able to easily modify Unsupported language or libraries Lack of skills No source code available!

Handling Legacy Systems

Incorporation Business as usual

Encapsulation Accessed from new system Adapters

Wrapper around the legacy system Adapters in new system

Technical Issues

Building maintainable software What features have we talked

about in code and documentation that help?

Technical Issues

Building maintainable software What features have we talked about

in code and documentation that help? Well documented code

Names, headers, style, … All the documentation

Architecture, design documentation, use cases, requirements, …

Today’s Tip

You need to make sure that when you turn in your final project, everything is consistent. Contract can’t be changed Other documents

For your enjoyment… “Yes, the world needs bankers and

artists and even professional athletes. They, among countless others, create the breadth of society and culture. But if you want tomorrow to come – if you want to spawn entire economic sectors that didn’t exist yesterday – those are not the people you turn to. It’s technologists who create that kind of future.” (Neil deGrasse Tyson, Director of Hayden Planetarium)