Making the Program FailCEN4072 – Software Testing – PPT3

Charles Powell

Overview

Making the Program Fail• As the title suggests, software testing involves executing the program

with the intent of making it fail. This section covers the setup of a program to be tested as well as the automation and isolation techniques of software testing.

• During any slide/topic, see the presentation notes within PowerPoint for further explanations as well as sources.

• Topics include the following: 3.1 Testing for Debugging 3.2 Controlling the Program 3.3 Testing at the Presentation Layer 3.4 Testing at the Functionality Layer 3.5 Testing at the Unit Layer 3.6 Isolating Units 3.7 Design for Debugging

Two Views of Testing• Validation Testing

The classical method of testing, where one seeks unknown problems, i.e. the tester expects valid results from the program.

A great deal of research is conducted on methods for uncovering these types of problems (see notes).

• Debugging When a known bug exists, debugging is used to correct the issue. This is the opposite of validation testing in that a known error exists, the bug,

and the testing process will revolve around the state of the bug and debugging process.

• This distinction between validation testing and debugging was first introduced by Glenford J. Myers in 1979 which forever divided the software development activity of debugging from verification.

Tests in Debugging

Create a test that

guarantees reproduction of the bug

Reproduce

Repeat and modify the test to omit

any unnecessar

y parts

Simplify

Dig deep in the code to isolate and correct the

faulty methods

Observe

Ensure the fix does not affect other parts of the

code

Validate

Repeat tests after releases to guarantee

the bug does not reappear

Regress

• The debugging process, discussed in later chapters, is as follows:

• The best way to follow this process is to automate it, making thorough tests possible at a frequent rate.

• Automation allows for repeatability, scalability, and isolation of test cases.

Automated Failure Induction• Automated tests can be used to isolate failure causes automatically,

including: Input – automatically test any number of possible inputs for your code.

Code Changes – when the error occurs, automation can point the programmer to the code being executed at the exact time of failure.

Thread Schedules – automation can step through threads, or a program queue of tasks (recursion, iteration, etc.), and find changes and errors in the schedules of these threads.

Program State – when a difference in the program state is the cause of an error, automation can detect which variables and values might be causing the error.

Challenges• Automation of testing presents a few difficulties that may seem

simple for a human for difficult for a computer to reproduce.

• For example, an automated debugger must synchronize its actions with the applications. If it tests without time control, the program may not have time to run or the automated tests might be too slow to be feasible.

• Another challenge is abstraction, where the automated debugger must not only provide input, but be able to interpret output dynamically as to make proceeding input valuable to the test case.

• Finally, the automated debugger must be portable in the sense that if the interface changes or the regression testing is being done on a new release, the automated debugger should require little effort to be re-used.

Interaction Layers• Especially when considering automation, it is key to break down a program into

three different layers of testing, two of which can only be accessed in testing through automation.

• Presentation Layer Handles the interaction with the user What is seen as the running application or program Accessed by automation and humans in testing

• Functionality Layer Encapsulates the functionality, independent of a specific presentation Considers the entire program code and flow/processes. Only accessed in testing by automation

• Unit Layer Splits functionality layer into multiple units These units work together to make the program work, but can be debugged

independently. Only accessed in testing by automation

Diagram Interaction Layers• A program can be tested (1) at the presentation layer, (2) at the

functionality layer, or (3) at the unit layer.

Interaction Layers

Presentation Layer• The presentation layer, or the environment perceived by the user, can

be broken down into three parts:

Low Level Interaction – a stream of mouse and keyboard inputs. These can easily be replayed and is the simplest form of automated testing.

System Level Interaction – when an automation test involves re-creating and controlling a whole system, usually done through a virtual machine, to inject hardware or system errors.

Higher Level Interaction – when automation tests control the program by means of graphical or ease-of-use interactions, making testing robust and user-friendly.

Presentation Layer Example

Low Level Interaction• In this example of low level

interaction, the script is merely interacting with the program by means of keyboard inputs.

• The send_xevents command simulates a user action, such as moving a mouse (giving coordinates), enter a key, waiting, and other types of standard user input.

Presentation Layer Example

System Level Interaction• In this example of system

level interaction, the script is is controlling the entire system, simulating powerups and the hardware of a real machine.

• While not shown, introducing hardware errors that are common for a runtime user and handling them make for a very robust application.

Presentation Layer Example

High Level Interaction• In this example of high level

interaction, APPLESCRIPT, known for its user-friendly language, interacts with Mozilla at a very high level, using references to GUI elements to simulate actions.

• In contrast to the low level example with Android code, where input had to be very specific using coordinates and the like, this high level example uses references making the testing very robust and portable.

Presentation Layer Discussion

Scripting Languages• Scripting languages can be used to perform all three levels of

presentation layer interaction.

• According to InfoQ, the top three scripting languages are PHP, Ruby, and Python.

• These three languages are highly rated for many reasons, including Open source, allowing for an organic development and not controlled in a

proprietary way Dedicated communities that support the users and end up developing

extremely useful techniques for using the code Ease of use, exception handling, extensibility and maintainability also

contribute to making the seemingly involved task of debugging a rather simple and enjoyable one.

• Other languages include: Actionscript, Flex, Javascript, Microsoft F#, Microsoft Powershell, Perl, PHP, Python, Ruby, VB Scrip

Interaction Layers

Functionality Layer• Instead of merely simulating program interaction as Presentation

Layer testing provides, it is much more useful for the program to provide an interface that is designed for automation.

• It is also sometimes impossible to simulate user interaction on the presentation layer with a program as interfaces become more and more advanced. Consider testing video games, virtual reality, or even car testing, where the user input is extremely advanced and could involve real-world implications even in testing.

• Furthermore, testing at the presentation layer means going through a great deal of results that may not be easily accessed. At the functionality layer, however, results are easily collected, accessed, and evaluated as the program was built to be tested with pertinent feedback.

Functionality Layer

Unit Tests• Using unit tests involves breaking down the code of a program into

units, typically done by the sheer design of a language (functions, libraries, modules, objects, classes, etc.), and debugging only the execution unit of which is causing the problem.

• Unit testing is only feasible if a bug can be isolated to a specific unit, and the testing can only be performed based on the behavior of a single unit.

• Unit testing has seen an increase in popularity as of late, mostly due to the concept of extreme programming where units are tested as early and as often as possible.

• The unit testing framework consists of: Setup of the environment Testing of the Unit Closing the Environment

Functionality Layer

Unit Test Framework• Setup the Environment

The framework sets up an environment for embedding the unit Frequently, a unit will require services of other units or the operating

environment. This part sets up the stage.

• Test the Unit Each possible behavior of the unit is covered by a test case in the framework. Each case first performs the operation( s) and then verifies whether the

outcome is as expected.

• Close the Environment The framework tears down the environment it just setup and tested. This means it brings everything back in the state encountered initially.

Unit Test Example

JUnit • In this example of a unit test, the

JUnit framework, an effective testing framework for numerous languages, is used to organize and conduct the automated test.

• As seen, the URL class is being tested with test case URLTest, a subclass of TestCase.

• Notice the setUp() and tearDown() methods responsible for the first and last steps of unit testing.

• This specific tests is for identifying equality and nonequality, using the assetEquals() method.

Unit Test Example

JUnit (cont’d)• In this continued example, we

see the GUI of JUnit running, where the box on the left shows a pass and the box on the right shows a failure.

• As suggested by extreme programming, unit testing should be done as early and often as possible, with some frameworks allowing tests to be run after each and every save of the program.

Unit Test Example

PyUnit• Built directly into Python,

PyUnit is a version of Junit ported as the de facto unit testing framework for the language.

• As seen in the code example, the structure of the code is the same as JUnit, but includes the Python conventions one would expect.

Unit Test Discussion

Circular Dependency

• Sometimes, programs depend on presentation to perform certain functionalities making it impossible to separate the two. This is referred to as circular dependence and can make testing and debugging extremely difficult.

• The question is, how can circular dependency be broken…?

Unit Test Discussion

Breaking Circular Dependency

• As seen above, one can adopt an inheritance scheme to break circular dependency.

• This means that the presentation level needed for functional dependency is upgraded to a superclass containing two separate possible instances, one for actual user presentation and one for automated presentation, or that of testing.

Designing for Debugging• These are some basic design concepts that will help any coder tackle

debugging: Good design and design methodology Consistent style (e.g., use C program idioms as much as possible) Boundary condition tests Assertions and sanity testing Defensive programming Avoid files that have a large number of functions and functions that have a

large number of lines Limit global data Checking tools These tips are explored more in the given source (notes)

• Also, one might consider the principle of reducing dependences by depending on abstractions rather than on details, as done in the Model-View-Controller architectural pattern (MVC)

MVC Architectural Pattern• The Model-View-Controller (MVC) architectural pattern decouples

functionality and presentation at the application level in order to reduce dependencies in debugging.

• To illustrate, consider an information system for an election day… We wish to illustrate election data in a a bar

and pie graph. We also wish to represent the data in a

spreadsheet that the user can manipulate. As shown in the figure, the goal is to separate

the functionality and the presentation by splitting it into two parts...

1. The model that holds the data and provides services to operate on the data (pictured here on top)

2. A number of observers that attach to the model and get notified whenver there are changes (pictured here on the bottom)

MVC Architectural Pattern (cont’d)• In this continued election

example, the model (coreData) has observers which are either views or controllers.

• Observers are divided into A view that is responsible for

displaying the core data in a specific way (in this example, a pie chart or graph view)

A controller handles input events (typically from the user) and invokes the services of the model.

• When a change does happen, all observers are notified of the change as to update themselves.

MVC for Debugging• Using an MVC architecture as previous discussed can have many

benefits for testing and debugging…

For testing, one can build and add new controllers that invoke services of the model— for instance, controllers that automate execution of these services. For debugging, one can register special views that automatically log all changes to the model.

For debugging, one can register special views that automatically log all changes to the model.

Finally, every observer and model can be examined and tested in isolation, thus reducing complexity.

General Design Rules• All software design models methods attempt to reduce dependencies

between units, using these same two principles:

High Cohesion – This is the basic principle that tells what to group into a unit. Those parts of a system that operate on common data (and thus depend on these

data) should be grouped together— typically, into some unit as supported by the design or programming language.

For instance, object-oriented design groups the state (i.e., data) and the functions that work on these data into classes and objects.

Low Coupling – This is the principle to reduce dependencies If two units do not operate on common data, they should exchange as little

information as possible. This principle is also known as information hiding, and is the key for

understandable, reusable, and extensible systems. The principle of low coupling also prohibits circular dependences, as they couple

all involved.

Tools for Automated Testing• In your experience, you may have found a testing interface, a development

environment, or testing software that you find comfortable to use for testing. It is also important to note that many popular code editors have debugging tools, much like the ones described in the previous slides, built into the software!

• JUnit, as mentioned previously, is a highly popular debugging framework to write repeatable tests. It has been ported to most languages and is extremely powerful both in a software and community sense.

• ANDROID script, per Dr. Dobb’s explanation, is a tool for recording and playing back scripts of events. In other words, it is a personal assistant that's smart enough to "do what you mean" when drudge work is waiting.

• APPLESCRIPTS, as mentioned, is a framework known for it’s user-friendly language and high level interactions in debugging.

• VBSCRIPT is a lightweight editor developed my Microsoft and modeled off Visual Basic, needing a host environment typically developed by Microsoft.

• …and many more! There are multiple environments to conduct the testing methods described in these slides, it is sometimes a mere matter of preference.

Making the Program Fail

Review• The two views of testing… for unknown and for specific failures

• Write tests to reproduce, simplify, observe, validate, and regress

• Challenges in automated testing include synchronization, abstraction, and portability.

• The three interaction layers Presentation Layer – the face of the program. Tested with at low level, system level, and high

level. Functionality Layer – how the code makes the program works, accessed only with automated

testing. Unit tests – breaking the functionality layer into isolated parts.

• Circular Dependency in unit tests, where the functionality depends on presentation, can be avoided with superclasses, where a separate class for automation testing is used.

• The Model-View-Controller (MVC) design architecture where the presentation and functionality is split into a core model and many views which reference the model.

• High Cohesion is the principle used to group units

• Low Coupling is the principle used to reduce dependencies.