1

CEN 4072Software Testing

PPT4: Reproducing the Problem

2

First things first: reproduce the problem

Recreating the environment – which is the setting in which the problem occurs. In a perfect world this would be done by using the computer in which the problem exists including customer machines; however, there are several factors which can make the solution difficult to obtain such as: For more detailed descriptions check the slide notes.

– Privacy – Cost of Maintenance– Travel Cost– Risk of – Ease of Development

3

Recreate the problem history – the steps needed to create the problem. This is this most likely solution to the previous slides issues. It allows for the developer to attempt to recreate the issue without using the physical machine in question. To do so the developer is to follow an iterative process containing the following steps which will be further explained in the next slide:

1. Start with your environment2. Add more circumstances3. Reproduce the execution

First things first: reproduce the problem cont…

4

Iterative reproduction

Start with your environment:Since it is impractical to have the exact system with the issue every time a bug is experienced a developer needs to recreate the environment as best as possible to recreate the bug for testing. First part to this is using the version of the software in question as indicated in the report. Then try and recreate the symptoms.Add more circumstances:If a simple creation of the environment did not allow reproduction of the issue then try and further recreate the system in question such as hardware, drivers, configurations, preferences, or anything else.Reproduce the execution:If both of the above are unable to produce the bug then further steps are needed. In order to reproduce the execution one must reproduce the program input.

5

Program inputs

Inputs for problem reproduction– Data – Least problematic. Files, databases.– User Interaction – Difficult to reproduce. Can

have minor variances which may be necessary to recreate the issue.

– Communication – Between computers, threads, or processes.

– Randomness – Every execution can be different.– Operating System – Provide additional services

that can interact with the software, but typically should not.

– Schedules – Scheduling algorithms could cause issue if defective. Not typical.

– Debugging Tools – Can interfere with execution in both ways that can mask, and bring forth issues.

6

Data

Get all the data you need – During attempt of reproducing an issue be sure to completely do so reproducing all symptoms. Including configuration, and registry files.

Get only the data you need – Some programs contain large amounts of data so be sure to only collect what is needed to avoid unnecessary data that could complicate debugging with erroneous data.

7

User interaction

Capture – The active process of recording the interaction with the program during execution, and writing it to a script. This is technically done through interrupt calls after the actual execution of input.Replay – The controllable reuse of recorded input data in script form. Can be used to repeatedly reproduce a set of data. Is nothing more than a stream of events.

8

Communication

Record since last checkpoint – In most cases it is not needed to record the entire communication since the start of the program. The only interest is in the reproduction of the issue. Which is most likely later transactions. So only this checkpoint (or state) is needed and not ones before it.Record last transaction – Similar to above the debugger is only concerned with the area surrounding the bug. So for example in a database where communication is recorded after every transaction we only want the last transaction. As it is the information that pertains to the issue.

9

Randomness

Program behavior in different runs – Essentially nondeterminism is a sought after feature in some program types. So by design it will run differently with each execution. However in some cases they are only pseudorandom and this allows for values to be reproduced.Check using random input – When the values are pseudorandom it is possible to reproduce them through random input until the desired output is found.

10

Operating system

Record and replay OS interactions – The actual recording as described earlier is done through inputs and the outputs from each input, and system calls. So for example a write() counts the number of written characters and read() reports what was actually ready in. Using this log replays are possible to recreate the environment, by using fake functions such as read(), and write() during replay.Traced interaction – As said above the produced logfile of the interaction using tools such as STRACE produce a series of inputs and their respective outputs with use of system calls. An example of a log can be found below.Challenges of replaying traced interactions – The sheer amount of data collected can be quite cumbersome. So use of checkpoints are preferred to monolithic logs.write(1, "Please enter your password: ", 28) = 28 read(0, "secret\n", 1024) = 7 write(1, "Access granted.\n", 16) = 16 exit_group(0) = ?

11

Schedules

Challenges regarding thread changes – There are several areas of difficulty surrounding scheduling. Problems arise when two threads simultaneously try and access the same critical section. Sometimes when this happens one of the threads will lock a resource until it gets all resources it needs which can result in circular wait and thus a deadlock. While this is only one example of problems that can arise out of schedulers they are very difficult to isolate or determine. Partly because the scheduler is a part of the Operating System which is abstracted to most developers who are not concerned with the Operating System itself.

Reproducing schedules – As schedulers and their algorithms can vary drastically this makes them difficult to give a set path of instructions to reproduce. Generally speaking to avoid the deadlock issue above algorithms can make use of different methods to prevent holding onto resources, and constant waiting by using something such as semaphores. This at least can alleviate the problems caused in a scheduler, but in real world application will almost never solve the problem completely. At this time the only ways to really determine a schedule are massively large random input tests, to uncover patterns and differences in execution.

12

Debugging tools

Code may fail outside the debugger only – There are times in which attempting to debug can actually ‘solve’ the problem for as long as one uses a debugging tool, but in no way actually solves the problem as it still remains after the tolls are done being used. This is an example of a heisenbug.Types of bugs:

– Bohr bug – A bug that can reliably be produced under well-defined set of conditions.

– Heisenbug – A bug that alters it’s behavior or disappears when debugging or probing is attempted.

– Mandelbug – A chaotic and complex bug that appears nondeterministic.

– Schroedinbug – A bug that is only found when someone is reviewing the source code and notices that the code should have never worked to begin with, but yet it does.

13

Isolating units

Capture and replay units – To isolate a unit with use of a control unit it is to attempt isolating arbitrary units from the environment. This is known as bottom-level units. Most of the time they only deal with services like communications and storage Reproducing bottom level units is far easier than reproducing the entire behavior of an application with that unit with use of capture and replay units. Needs unit control layer – As described above a control layer is needed to isolate the units. In a sense a control layer is a generalization of the STRACE tool (textbook).

14

Isolated units

The following are examples of isolating units with use of bottom level units to reproduce a problem in these types.

Compilers – Record and restore the intermediate data structures.

Networking – Record and replay the basic communication calls by the application.

Databases - Record and replay from SQL transactions produced by the application instead of executing the application to do the same. As an example a small script while using TSQLt to assert equality in a database will be on the next slide.

15

Isolated units cont…

EXEC tSQLt.NewTestClass 'testFinancialApp'; GOCREATE PROCEDURE testFinancialApp.[test that ConvertCurrency converts using given conversion rate] AS BEGIN

DECLARE @actual MONEY; DECLARE @rate DECIMAL(10,4); SET @rate = 1.2; DECLARE @amount MONEY; SET @amount = 2.00; SELECT @actual = FinancialApp.ConvertCurrency(@rate,

@amount); DECLARE @expected MONEY; SET @expected = 2.4; --(rate

* amount) EXEC tSQLt.AssertEquals @expected, @actual;

END; GO

Further explanation can be found on the source page with URL..

16

Log as a program

One of the easiest ways to a log file of a program is to make the log a stand alone program that is executable that asserts the output with the inputs given. To do so the code must be overwritten with redefined functions. The inputs of the log function are below.#include "Map.h"

#include <assert> int main() {

Map map; map.add("onions", 4); map.del("truffels");assert(map.lookup("onions") == 4); return 0;

}

The rewritten redefined function for add is below with use of clog which essentially outputs the key and value in the function to an output log file.

void ControlledMap::add(string key, int value) { clog « "map.add(\"" « key « "\", " « value « ");" « endl; Map::add(key, value);

}

17

Logging interaction with the OS

A quick example of the usage of clog is below; however, it should be said that clog will display to terminal just as cout will so it should be used appropriately and redirected to differentiate the two (same with cerr). The below example attempts to open a file that is unable to be opened by design to force a failure which then uses the clog message over displaying the file.

18

Sometimes a unit may work in isolation but not within the application itself. This is typically attributed to some interaction that is not yet properly monitored. Such as one of the following.

Variables – Are much easier to observe with mock tools which allow for easier control and observation while attempting to alter variables the application can access with units.Time - Some problems require a specific amount of time to elapse between function calls to occur. It is helpful to record and replay the time intervals.Other units with possible dependency – In other cases units may depend on some other resource that is controlled by the application. In order to capture and restore the state of these units decencies must be broken. This can be done by isolating the units.

Other interaction types

19

Tools

WINRUNNER – Popular set of tools used for recording and replaying user inputs. Allows for automation using test scripts. An example is Revirt. Which replays an entire interaction with a machine. Checkpointing Tools – a variety of tools that allow for ‘freezing’ a running process for later resuming. An example would be a tool called ReCrash which records and replays crashes for Java applications.Microsoft Fakes – Utilizes two different technologies referred to as shim, and stubs. Shims allow for you to isolate components under test from the environment. Similar to a log as a program. Stubs are very similar to mock objects where they take place of a piece of code of a component. This allows for testing even with incomplete code with functions that have yet to be written.

20

Tools Cont…

Isolator++ - A tool that creates mock Windows API calls that works for testing legacy code without need to change it.

21

"Free Image on Pixabay - Microphone, Record, Play, Device." Free Vector Graphic: Microphone, Record, Play, Device. Web. 19 Sept. 2015.

"Free Image on Pixabay - Intercom, Telephone." Free Vector Graphic: Intercom, Telephone. Web. 19 Sept. 2015.

"TSQLt Tutorial." TSQLt Database Unit Testing for SQL Server. 5 May 2011. Web. 21 Sept. 2015.

http://www.math.hkbu.edu.hk/parallel/pgi/doc/pgC++_lib/stdlibcr/clo_8238.htm clog examplehttp://www.typemock.com/isolatorpp-product-page

https://msdn.microsoft.com/en-us/library/hh549174.aspx

Sources