Department of Computer ScienceFaculty of EngineeringLund University

Master Thesis

Development of Automated Testing Tools for Java MIDlets

Authors:Love LöfdahlManfred Nilsson

Supervisors:Michael Lundahl Karnerfors, instructor auSystemsBjörn Regnell, examiner

Abstract

We have preformed our master thesis at auSystems Sweden South in Malmö. The goal of the thesis was to develop a testing tool for automated tests of Java MIDlets based on the Mobile Application Framework. Java MIDlets are Java applications developed in Java 2 Micro Edition, a stripped version of the Java runtime environment, and Mobile Application Framework is a framework for developing MIDlets created by auSystems. To make the automated testing possible we created a testing framework. The framework is made up of a scripting language for describing the tests, a parser to pars the test scripts, an implementation of the OBEX protocol to transfer files to and from a mobile unit, a back end support for storing the test results in a database and a graphical user interface to let the user interact with the testing framework.

The difficulties with automated testing in general, testing on mobile platforms in specific, and our solution to these problems is what we will discuss in this report.

Table of Content1 Introduction 5

1.1 Disposition 51.2 Abbreviations and Acronyms 5

2 Background 72.1 Objective 72.2 Problem Specification 7

3 Method 83.1 Prestudie 83.2 Implementation 83.3 Evaluation 83.4 The evaluation criteria 9

4 Verification & Validation (V & V) 104.1 Testing 10

4.1.1 Unit Testing 114.1.2 Integration Testing 114.1.3 Release Testing 114.1.4 Stress Testing 114.1.5 Black Box and White Box Testing 11

4.2 Test Specifications 124.3 Test case design 12

5 Tools and Measurements 135.1 Testing maturity 135.2 Software Quality Characteristics 135.3 Tools of the trade 14

6 Automatic-Testing 156.1 Test Tools 156.2 Functional Decomposition 156.3 Data driven Testing 166.4 Using wrappers to increase flexibility 166.5 Involving the developer 176.6 Goals when introducing Automation 176.7 Automatic Testing done manually 17

The TANT framework 186.8 The Manfred and Love scripting Language 18

6.8.1 The MaL-code parser 186.8.2 Keys and Delay 196.8.3 Standard Functions 19

6.9 A Typical Test Script 206.10 Recording a Test Case 21

6.10.1 Discussion about Keyboard Logging 216.11 Configuration Files in TANT 22

6.11.1 Test Case Recorder Configuration File 226.11.2 Emulator Playback Configuration File 226.11.3 Mobile Unit Playback Configuration File 236.11.4 How the Configuration Files Overlaps 23

6.12 Executing a Test Case on Emulator 236.12.1 What is an emulator ? 236.12.2 How the playback is done 246.12.3 The Virtual Keys 24

6.13 Executing a Test Case on Device 24

6.13.1 Serial Connection 246.13.2 AT-commands 256.13.3 OBEX 256.13.4 How the Playback is done 26

6.14 Testing Without AT-Commands 266.14.1 TestEngine 266.14.2 MAF 276.14.3 The TestMIDlet 276.14.4 How the Execution is done 27

6.15 Result Handling 286.15.1 The Database Result Handler 286.15.2 The Temporary Result Handler 286.15.3 Screen Shots 286.15.4 Execution Loggings 29

7 Discussion and Conclusion 307.1 Result 30

7.1.1 Validity 317.1.2 Result Discussion 317.1.3 Will TANT be Used? 31

7.2 Limitations 327.3 Future Improvements 327.4 Conclusion 33

8 Bibliography 34

Appendix A Layout of TANTAppendix B Test Case Recorder Configuration File for Sony-Ericsson emulatorAppendix C Playback Configuration File for Sony-Ericsson K750 emulatorAppendix D The Mobile Unit Configuration FileAppendix E Test SpecificationAppendix F TANT

1 Introduction

The following chapter will give the background to this master thesis. It will specify the problems that we tried to solve, the specific demands on our solution and present all the abbreviations and acronyms that we use.

1.1 Disposition

In chapter 1 the background to this master thesis will be introduced and the acronyms and abbreviations found in the paper will be presented. Chapter 2 will present the objectives and the problem specification. Chapter 3 describes the method we have used under the course of the master thesis. Chapter 4 will discuss basic testing theory and chapter 5 deals with testing maturity theories and software quality measurements. Chapter 6 will discuss automatic testing. Chapter 7 will introduce the TANT framework and how it performs automatic testing. Chapter 8 presents the results of our evaluation of TANT and how well it solves the problems specified in chapter 2. Chapter 8 also contains the concluding discussion and ideas about future improvements of TANT.

1.2 Abbreviations and Acronyms

AST: Abstract Syntax Tree. A tree data structure that describes parsed source code. Each node in the tree is token in the programing language.

AT-command: Attention command. A set of commands that can be sent to the internal modems of a mobile unit.

CASE: Computer Aided Software Engineering. Tools that helps the development of software.

CMM: Capability Maturity Model.

GUI: Graphical User Interface. The part of a system that the user interacts with.

MAF: Mobile Application Framework. A framework used to develop midlets created by auSystems Sweden South in Malmö.

J2ME: Java 2 Micro Edition. A stripped version of the Java Runtime Environment used to execute java programs in environments with small resources such as cell phones.

JNI: Java Native Interface. An interface to call methods written in C or C++.

Keyboard Hook: A keyboard hook registers with the Windows operating system.

5

When a keyboard message occurs at the input it sends a keyboard message to a callback method.

MaL-code: Manfred and Love scripting language. A scripting language developed in JavaCC for the TANT testing environment.

MIDlet: A program developed to run under J2ME. Common to all MIDlets are that they extend the super class MIDlet.

OBEX: Object Exchange Protocol. A protocol used to transfer objects (usually files) between a server and a client.

OS: Operating System. A system that provides a foundation of basic routines and manages hardware resources.

RGB: Red-Green-Blue. A way of describing colors by specifying how much red, green and blue it contains.

SUT: Software Under Test. The software that is tested during a test.

SVN/CVS: Subversion/Concurrent Versions System. Version control systems used to keep track of different versions of programs.

TANT: Testing Application for (Non)intrusive Testing. A framework for automated testing on mobile telephones. The practical result from this master thesis.

TIM: Test Improvement Model.

TMM: Testability Maturity Model.

TMAP: Test Management Approach

WAP: Wireless Application Protocol. A protocol for Internet access through mobile units.

6

2 Background

auSystems Sweden South (auSSS) are developing products for Java 2 Micro Edition (J2ME). Although all J2ME compatible devices can run J2ME MIDlets, they may still look and act different on different platforms. This is be course every manufacturer delivers there own implementation of the runtime environment.

To handle this problem auSystems have developed a framework called Mobile Application Framework (MAF). By using MAF auSSS can shorten the development time for MIDlets and give them the same look and feel on different versions of J2ME. To further shorten the time it takes to deliver a product auSystems wants to automate their testing efforts.

There are many different product for automated testing on the market. Some of them are very expensive and most of them are not able to handle the difficulties that are specific for testing on mobile platforms. In this report we will discuss our implementation of a framework for automated testing on mobile phones.

2.1 Objective

The objective of this master thesis is to implement a framework for automated testing of MDIlets developed with MAF. To do this we have to solve some problems that are specific for testing on mobile telephones but also some problems that are common to all forms of automated testing. Some of the problems will be presented in this chapter and our solutions will be discussed in the rest of the paper. An evaluation of our solution will be found in chapter 3 and 8 and a general discussion about the resulting framework can be found in chapter 7.

2.2 Problem Specification

One of the biggest problems we faced were that there are so many different platforms to test on. One MIDlet might have to look and behave the same on 10 different phones and that have to be tested. To be able to test on so many platforms the resulting framework had to be very flexible. Another of the problems, concerning automated testing in general, were that it had to have an edge over manual testing. Or else there would not be any use of introducing the automated testing.

7

3 Method

This master thesis have been preformed in three stages. First we consulted literature about testing, then we used that knowledge to created a testing framework for testing of MIDlets and finally we evaluated the results by testing our framework.

3.1 Prestudie

There are many books and papers written about testing and automatic testing. To increase our knowledge in the subject we preformed a text studie. Most of the literature that we studied promote their own approach to automated testing of software but they all have some common points. One of these points are the importance of having a scripting language to describe the execution of a test that is easy to use. Another important point is that the scripts can be edited manually and to not rely solely upon recording and playback. It is also very important that the company supports the introduction and continued use of the automatic testing method. If not the initial extra work load to get used to the system might stop the testers from using it.

3.2 Implementation

The Test Application for (NON)Intrusive Testing (TANT) framework described in more details in chapter 7 were mostly developed in Java with a few details implemented in C. We also used JavaCC to write a parser for the Manfred and Love Scripting Language (MaL-code) described in more details in chapter 7.1. To handle serial communication we used the RXTX1 library developed by Keane Jarvi published under the GNU Lesser Public License agreement. The Java code were developed using Eclipse and using SUN Microsystems Java 6 SDK and SonyEricssons J2ME SDK. The C code was developed using Dev-C++

3.3 Evaluation

In order to evaluate our work we had to set up a couple of test cases. These tests were run on the following platforms.

• Sony Ericsson K750

• Motorola J2ME Emulator

• Sony Ericsson J2ME Emulator

The tests were preformed in three different ways on all of the platforms.

1 More information can be found at the homepage, http://www.rxtx.org/.

8

• Testing performed Manual: The tests were preformed manually according to the test specification in appendix E. No other tools than the software for uploading the MIDlet to the mobile platform and the emulators were used.

• Testing performed with the aid of TANT: The tests were preformed with the aid of the TANT framework and according to the test specification in appendix E. The MaL-code scripts were written and executed according to the specifications in Chapter 7. No other tools then TANT, a program for viewing the screen dumps and a text editor for writing the MaL-code scripts were used.

• Testing performed with the aid of TANT using TestMIDlet: The tests were preformed using the TestMIDlet, described in chapter 7.8, according to the test specification in appendix E. The TestMIDlet was upploaded using tools suplied by the cell phone manufacturer. The execution of the tests were controlled by the TestMIDlet. No other programs but a program for viewing the screen dumps were used. TANT could have been used to upload the TestMIDlet but there was no need to use it.

3.4 The evaluation criteria

The test results were evaluated by how long time it took to perform the tests, including the time to write the test scripts, and how many tests that generated the right results.

The total time is not the only time that have to be considered. When a test is executed automatic there is no need for supervision and the time while the test is preformed can be used to other things. Therefor the total time of the test does not tell the whole truth. The actual time that have to be invested in preforming the test is also evaluated.

9

4 Verification & Validation (V & V)

All software needs to undergo some sort of testing to know that the right product is being built (Validation) and that the product is being built right (Verification) (Sommerville, 2007). We have in our work chosen to focus more on the verification part and leave the validation up to others.

The goal of the verification is to establish whether or not the software is fit for its purpose, as different projects have different budgets and quality demands. This means that the level of confidence in that the product performs according to specifications needed will vary from project to project (Sommerville, 2007).

Verification can be done through static methods such as reviewing the code or through automatic analysis of the code. Today most modern compilers include some form of code analysis. There are also dynamic methods that involve running the code to generate output data. This is generally refer to as testing. Running the code can be done manually or automatically (Sommerville, 2007).

Code reviewing has been found to be quite effective in discovering errors early. The cost for correcting an error discovered early is lower than the cost for similar errors discovered once the code is runnable and submitted for testing. Code reviewing also have the added benefit that it can be used on code that depends upon unfinished code with no additional costs. It would also be possible to review unfinished code however this would be far less cost effective. (Sommerville, 2007)

A successful test will reveal a defect in the code. After the error have been corrected the code needs to be tested again to ensure that no new errors have been introduced into the code. Retesting will also discover any errors that been previously obfuscated. (Sommerville, 2007)

4.1 Testing

Most test are designed to detect if the code contains errors or if the system deviates from the specifications. There are measurements such as code coverage to measure how well the code has been tested, even so it is not possible to conclude that the code is faultless, only that no errors were found. (Sommerville, 2007)

Good testing practice dictates that the person responsible for testing should not be the same person doing the development. For good reasons there is an inherited danger that the person who developed the code will write test cases to test what is working, rather than to try and discover if it adheres to the specification. Good testing practice also dictates that tests should be written for all code. The best approach is to write test cases before writing any code.

10

4.1.1 Unit Testing

Normally the developer should write a test case for each class or similarly small unit of code. This practice of writing tests for each class is called unit testing.

4.1.2 Integration Testing

Building upon unit testing is integration testing or incremental testing. In integration testing, tests are preformed to ensure that different parts of the developed system or program interacts as planned. In preferably small steps functionality is added to the system during integration testing, retesting it after each incrementation. Not only should the new functionality be tested but also all previous tests should be rerun, this is called regression testing. Since regression testing is a repetitive task and has a given output, it should yield the same output as before the new functionality was added, it is well suited for being automated. (Sommerville, 2007)

4.1.3 Release Testing

Before releasing a product to the customer, steps should be taken to ensure that it do not fail during normal use and that it produces desirable results. Usually this is done as a set of tests to test the function of the system not taking into account the implementation. During these test it is common to attempt to break the system for example by choosing inputs that are more likely to yield unwanted output. Examples would be testing null or zero as well as other boundary values. (Sommerville, 2007)

4.1.4 Stress Testing

Stress testing is preformed by applying a load outside specifications and forcing the system to break down. This kind of testing can be used when confidence that the system will perform according to the specifications have been established.

Stress testing can be used to determine how the system behaves in abnormal situations. It is preferred that although the system may crash it does not cause any additional damage in the form of corrupted data or malicious control signals.

Response times might also be dependent of the load the system is under. Worth considering is also the distribution of load across different types of tasks.(Sommerville, 2007)

4.1.5 Black Box and White Box Testing

Black box testing is when knowledge of internal implementation details are not considered when designing test cases and instead relies solely upon the specifications. The opposite is called white box testing. White box testing is an attempt to make an educated guesses about what might be incorrect based upon the knowledge of how it is

11

implemented. For example some types of programing tasks are know to be more inclined towards certain types of errors. (Pol et al., 2002)

4.2 Test Specifications

A test specification for a certain test case should include at least some starting position, what actions to be taken as well as some predication for the outcome of the test. It is of course important that it is possible to repeat the test case over and again. (Pol et al., 2002)

4.3 Test case design

One approach for designing test cases is requirements-based testing. From each requirement in the specification a test or or a set of tests are formulated that verify that the specification is properly implemented. (Sommerville, 2007)

By dividing similar inputs into groups or equivalence partitions and also making the assumption that if a test case holds for a sub-section it holds for the whole set, the number of tests can be drastically reduced. (Sommerville, 2007)

An other structural approach is path testing where all individual paths available through the system are found and tested. The number of paths can easily be calculated (for programs without goto statements) it is simply one more than the number of conditions in the program, note that compound conditions have to be counted multiply times. This is referred to as the cyclomatic complexity.(Sommerville, 2007)

12

5 Tools and Measurements

To be able to accomplish even the most rudimentary automation, some sort of test tool must be used. The tools can be developed in house or an of the shelf solution could be purchased . It is important to view the tool as a independent software project in the sense that it to needs to be planned and tested just as any other software project. (Zambelich)

5.1 Testing maturity

Software testing manual or automated is a process, and there is a need to deal with the necessary administration. There are several methods directed at software testing which aim to make this more effective, Testability Maturity Model (TMM), Test Improvement Model (TIM) or most notably Capability Maturity Model (CMM). They are all maturity models off sort. They classify the process into different levels of maturity dependent of how the process is facilitated. On lower levels there are no requirements and most is done ad hoc. Steping up in the ladder there are more rigorous demands for documentation and test specifications. As well as the possibility to follow the process from the specification of the requirements through development and testing. (Thomas et al 1997, Mitchel H 1994, Daniel et al., 2005)

The TMAP approach argues that it is important that the testing environment in which a test tool is introduced has reached a certain degree of maturity or the testing can prove to be contra productive. (Pol et al., 2002) Indeed many argue that the most important factor in making a successful attempt at introducing automatic testing is that it has enough support from the management and that the initially larger workload is handled appropriately.

5.2 Software Quality Characteristics

In order to have any understanding of what quality that have been achieved some measurements have to be established. There is a ISO standard defining six software quality characteristics. (Pol et al., 2002)

ISO9126 gives • Functionality The capability of the software to provide functions that meet

stated or implied needs when the software is in use under specified conditions.

• Reliability The capability of the software product to maintain a specified level of performance when used under specific conditions.

• Usability The capability of the software product to be understood, learned, used and attractive to the user under specified

13

conditions

• Efficiency The capability of the software product to provide appropriate performance, relative to the amount of resources used, under stated conditions.

• Maintainability The capability of the software product to be modified. Modifications may include corrections, improvements or adaption of the software to changes in environment, and in requirements and functional specifications.

• Portability The capability of the software product to be transferred from one environment to another.

5.3 Tools of the trade

The same measurements apply to testing tools although different meaning and weight can be applied to some of them. In Automation snake oil James Bach lists some factors specifically for Automatic testing. Hence we could add to the list from chapter 5.2.

Non-intrusiveness. How well does the tool simulate the user, or is it necessary to make adjustments to the code. (Bach, 1999)

Also worth noting is that when it comes to automatic testing speed is not very important when the test tool, usually, can not be faster than the Software Under Test (SUT). Reducing the need for performance tuning (Bret, 2001).

14

6 Automatic-Testing

While developing software for automatic testing one should consider the fact that, test tools are software and as such should be subject to the same quality assurance as other software. There should be version handling, specifications and testing as well as follow up. Otherwise trust can not be built for the results produced by the test tool. (Bret, 2001)

6.1 Test Tools

To be able to rationalize testing different tools can be used to help organize and even execute some of the tasks involved in the testing process. To support version handling there are Subversion/Concurrent Versions System (SVN/CVS) systems. There are bug tracking systems to help monitor discovered bugs and keep track of which bugs have been fixed, or sometimes more importantly to assign responsibility for fixing the bugs. If Computer Aided Software Engineering (CASE) tools have been used during the development there is usually little overhead in generating test coverage and checking correctness. To aid in the bug fixing there are debuggers who can track systems states during execution. Other tools can be used for generating input data for running test which require large amounts of input data, or tools that through some other method generates output data that are correct given a certain set of inputs. Later it is possible to compare that previously generated data to the output from the SUT. Here we can in turn make use of tools for comparing data.

If testing is commenced before all the code has been written the use of stubs and drivers could be used in order to facilitate the executing of the code. If the hardware are not available simulator could be used to test the code, or if stress tests is performed a tool might be used to simulate many simultaneous users. Even when the code can not be run it would still be possible to analyze it through the use of static analyzers. Regression testing can be facilitated through the use of record and playback tools.

And if it all becomes hard to keep track of there are tools to help with the planning and keep track of what progress have been made. (Pol et al., 2002)

6.2 Functional Decomposition

Functional decomposition is a method based on the old device divide and conquer. The execution of a test is divided into separate actions. Each action is defined by a subroutine script. These are in turn used by business scripts which describes a business function. One or more business scripts is included in a test case. On the top there is a driver script which performs the necessary set up and tear down of the test environment as well as initiates the different test cases in an orderly fashion. Data should be kept entirely separated from the scripts. This approach will lead to a library of test case scripts. The positive is that it is well suited for reuse on the negative side it requires

15

maintenance. If the business scripts were defined using only subroutine scripts we should only have to update the subroutine scripts to accommodate any changes in the SUT leaving the rest of the structure intact. (Zambelich)

6.3 Data driven Testing

If the test cases are well written these could easily be turned into scripts for automatic testing using so called action words. As with functional decomposition this method builds upon the base of a library with low level functions. While the test case is outlined in a standard spread sheet editor with columns for data entered and result expected. Action words would be added as command words to be interpreted as calls to sub scripts. Then it would be possible to export the spreadsheet into, for example tab delimited data, something most spreadsheet editors support. With this technique there would not be any significant need for technical competence concerning test automation amongst the testers. (Zambelich)

6.4 Using wrappers to increase flexibility

By adding an additional layer in the form of wrappers around for example each command of the test tool consisting of nothing other than a call to the wrapped function. This can later prove to be very useful for amongst other things stress testing and debugging. It makes it possible to add exception handling when needed. To take advantage off a new feature in the test tool or to circumvent a bug in the test tool and in the case that you decide to switch to some other test tool it might ease the transition.

16

Figure 1: Building a library of subroutines using functional decomposition

Figure 2: Data driven testing builds testcases using specific action words

(Cam, 1997)

6.5 Involving the developer

Many pitfalls can be avoided if the developers are given opportunity to partake in the planning of the testing. Simple things as standardized formated output and error messages as well as developing and including more intricate hooks for the automated testing can make the automation effort easier or more effective. (Cam, 1997)

6.6 Goals when introducing Automation

Automatic testing can have a lot of different goals. So naturally the outcome of projects must be judged against those goals. Automatic testing is not a universal remedy for testing.

Once the initial workload with constructing and setting up everything for automatic testing have been taken care off the effort for each run is very small compared to manual testing. Hence testing could be done more frequently.

Given the right conditions automatic testing can reduce costs of testing by reducing manual labor. Regression test, smoke tests and other test that contain many repetitive moments are particular suited for automatic testing. However there are task that can be difficult to automate due to technical considerations or simply is not worth it given the time it takes to automate contra the time it takes to perform manual testing. If automation is decided upon it is important to allocate enough time for it to be successful. Expectations of early payback from introducing automatic testing will most often not be fulfilled.(Elfriede, 1999) Payback could realistically be excepted in a second testing cycle.(Linz and Daigl, 1997)

Automatic testing can overcome some quality issues related to manual testing. For good or bad a machine does what it have been told and nothing else. Humans might make mistakes or introduce unwanted variations. There are also situations that can be hard to create and recreate manually, such as stress testing and most testing that involve timing issues. (Mitchel H, 1994)

If implemented correctly automatic testing can improve the confidence in the testing performed. Given that there have been accurate logging of what has occurred. One must then review or otherwise validate the logs produced. An alternative could here be to incorporate both the error logging systems as well as some sort of scanning tool into the product, for benefit both for the customer and tester. (Bret, 2001)

6.7 Automatic Testing done manually

Some aspects of testing is naturally more suited for automation others are not. When in doubt whether to automate or not it is probably a good idea to prepare for automation and postpone the evaluation. (Buwalda et al., 2002)

17

The TANT framework

Testing is the most common way to verify that a software does what it is supposed to do. To preform a test there must first be a test specification. The test specification describes what to test, how to test it and what results that are expected from the test. When this is preformed manually the tester is responsible for the execution of the test. In TANT the tester is replaced by a script that describes the execution of the test, step by step. TANT is not responsible for gathering test data or evaluating test results. That part of the test must be represented in the SUT. If a test is preformed to verify that a screen looks the same way on 3 different mobile phones, TANT may be used to step the program to the given screen but the actual capturing of the screen must be done by the SUT.

Some of the building blocks of TANT is described in the following chapters.

6.8 The Manfred and Love scripting Language

The MaL-code is a script language that is used to describe how a test in TANT should be executed. The MaL-code language includes keys, delays and standard functions. The actual compiler was written in JavaCC. The result of the parser is an Abstract Syntax Tree (AST) from witch a command list is built.

6.8.1 The MaL-code parser

The MaL-code parser was written in JavaCC. JavaCC is an open source, top-down compiler constructor with Java like syntax. Since most of TANT was written in Java, JavaCC was the obvious choice. It is very easy to integrate compilers written in JavaCC in other Java programs since the result are Java class files.

The result of the MaL-code parser is an AST. Once the AST have been created it can be traversed just as any other tree data structure. In TANT the AST is traversed recursively and a command list is generated (see figure 3)

18

6.8.2 Keys and Delay

The pressing of a key is represented in the MaL-code as [key, time], where key is the name of the pressed key (ie UP, DOWN, LEFT, RIGHT, and so forth), and time is the time that the key is pressed down in milliseconds, time is optional. Keys are used to step through an execution of the SUT.

Delays are used to get the execution to wait. This may be used to wait for a response in the program or to give the mobile unit or emulator time to preform the actions the script tells it to do. A delay is described by Delay(time)

6.8.3 Standard Functions

The standard functions are functions that have specific functionality associated with them in the MaL-code language. There are ten standard functions where delay have already been described. The rest of them follows here:

• Identification(). Identification is used to identify what mobile unit the test is run on.

• StdDelay(time). StdDelay changes the standard delay from the default value of 500 milliseconds to the value of the input parameter time. A standard delay is a delay that is automatically added to the command list if no delay is specified between two commands. Time is given in milliseconds.

• StdKeyPressed(time): StdKeyPressed changes the standard time to press a key, if no time is given, from the default value of zero to the value of the input parameter time. Time is given in milliseconds.

• WaitForFocus(time): When TANT executes a script on an emulator it must wait for the Operating System OS to give the emulator focus. WaitForFocus changes the default wait time from zero to the value of the input parameter time. Time is given in milliseconds.

• UploadProgram(path): UploadProgram uploads a program to a mobile unit

19

Figure 3: The MaL-code scripts is parsed and turned into a list of commands

using the Object Exchange Protocoll (OBEX) or starts an emulator with the program. Path is where the program is located.

• GetResult(path): GetResult gets all files in the given path and moves them to a subdirectory to results named as the test suit id. If the execution is done on a mobile unit the files is downloaded using OBEX. If the script is run on an emulator the files are moved by OS specific commands. Path does not have to be specified for the emulator so path is optional.

• Setup(): Setup is a mix between a standard function and a key. Setups functionality is not defined by the MaL-code but it depends on the current configuration file. What Setup() should do might depend on what platform the script is running so that is why it can not be defined in the MaL-code. See chapter 7.5 for more information on the configuration files.

• Sub(path): The last standard function is Sub. Sub is used to call an other test script. This allows code fractioning and code reuse. See chapter 6 for more on data driven testing and function decomposition. Path is either the full path or just a name of a script in the testCases sub directory.

By combining keys, delays and standard functions a MaL-script is created as seen in figure 4.

6.9 A Typical Test Script

A typical test script can be seen in figure 5 and will be explained in this chapter, line for line.

All MaL-code scripts begins with script { and ends with }. This is to help the parser understand where the MaL-script begins and where it ends.

Setup() makes the mobile platform ready to run the script and if the script is executed on an emulator the Setup() command is ignored.

20

Figure 4: A MaL.-code scripts constitutes of keys, delays and standard functions

StdDelay(1000) sets the standard delay to 1 seconds. If now standard delay is given for the script it is set to 500 milliseconds by default.

UploadProgram(path\to\file.jad) uploads file.jad to the mobile unit. The mobil unit then downloads the jar file via Wireless Application Protocol (WAP) or internet from the url specified in the jad file. If the script is executed on an emulator, an emulator is started and the jar file is downloaded from the url specified in the jad file via internet.

[LEFT_SOFT_KEY,100] will simulate the left software key being pressed for 0.1 second and [END] will simulate the return key being pressed for the time set by StdKeyPressed(1000). Since there is no delay command between the two keys a standard delay of 1 seconds will be added when the tree is traversed.

GetResult(path) will download everything in the directory path if the script is executed on a mobile unit. If the script is executed on an emulator the OS will move all the files in the directory specified by the settings for the emulator.

6.10 Recording a Test Case

TANT includes the possibility to record the execution of an emulated MIDlet. This is done by adding a keyboard hook to the Windows OS and record the keyboard events that the hook generates. The result from the recording is a MaL-code script file.

Since only the keys that actually corresponds to a key on the emulator should be recorded, TANT only logs the keys that corresponds to a test case recorder configuration file.

There are two modes of recording in TANT, simple and advanced. In advanced mode, the time of the keystroke is recorded. In simple mode only the key is recorded.

6.10.1 Discussion about Keyboard Logging

Some forms of test case recording requires a working copy of the software to be tested to be running. In such cases there might not be much use of a recorder since the program is already up and running. In some situations however there is a need to generate a test script very quick and then recording can be a valid alternative.

21

Figure 5: A short MaL-code script

The recorder in TANT does not require an emulator to be running. As long as the recording is active, all the keystrokes that should be logged, are logged. So recording a script might be seen as an alternative to typing the whole script from scratch. See chapter 7.4 for more on record and playback.

6.11 Configuration Files in TANT

There are three types of configuration files in TANT. They are used to make TANT as open to changes and compatible with as many testing platforms as possible. The three types are

• Test case recorder configuration file: The first type of configuration file is used to make sure that only keys that corresponds to a certain emulator is logged.

• Emulator playback configuration file: The second type of configuration file is used to go from a MaL-code script to virtual keyboard strokes. This is needed since different emulators might have different key setups.

• Mobile unit playback configuration file: The third type of configuration file is used in same way as the emulator playback configuration file but for mobile units. This is needed since the available Attention Commands (AT-commands) may differ from manufacturer to manufacturer.

6.11.1 Test Case Recorder Configuration File

The test case recorder configuration file is a type of configuration file used by the test case recorder in TANT. It maps a virtual key code value (a code specified by the OS) of the pressed key to a text string.

Only the keys present in the configuration file are logged to the script. If there were no kind of filter, all keys would be logged to the MaL-code script and it would be very hard to read.

An example of a configuration file can be seen in appendix B.

6.11.2 Emulator Playback Configuration File

The emulator playback configuration file is used to map the keys and commands in a MaL-code script back to virtual keys. This is important because a MaL-code script should be able to run on many different emulators.

The content of the configuration file should be every key that the emulator recognize mapped to the virtual key that should be sent to the emulator.

An example of a configuration file can be seen in appendix C.

22

6.11.3 Mobile Unit Playback Configuration File

The mobile unit playback configuration file is used in the same way as the emulator configuration files. But instead of MaL-code keys being mapped to virtual keys the keys are mapped to AT-commands. There must also be a mapping from OBEX to the AT-command for going to OBEX mode on the mobile unit.

An example of a configuration file can be seen in appendix D

6.11.4 How the Configuration Files Overlaps

The configuration are what binds the different part of the TANT framework together (see figure 6). For recording and playback to work, some rules must be followed. The right hand side of a line in a test case recorder configuration file must correspond to the left hand side of a line in an emulator- or mobile unit-configuration file.

VK_F1=LEFT_SOFT_KEY must therefore correspond to LEFT_SOFT_KEY=VK_F1 or LEFT_SOFT_KEY=AT+CKPD="[". If the specifications are not followed the recorded keys will not be recognized by TANT.

6.12 Executing a Test Case on Emulator

The following chapter will discuss how TANT executes test cases on an emulator and give a short introduction to what an emulator is and why they are used.

6.12.1 What is an emulator ?

J2ME have the same compatibility claims that regular Java has, build for one, works on all. But as mentioned before, this is not entirely true for J2ME. So when a MIDlet is

23

Figure 6: How the different configurations files overlaps

developed it is important to test it on as many platforms as possible. There are however a lot of different cell phones on the market and to buy them all would be very expensive. The solution to this problem are emulators.

An emulator is a programs that is built to resemble the J2ME runtime environment on the unit it emulates. So with an emulator it is possible to test on a lot of different units without having to buy them.

6.12.2 How the playback is done

Once the MaL-code parser have parsed the script it returns it to TANT as a command list. TANT then iterates over the command list. If an Uploadprogram command is found, the emulator which have been selected for this test is started. If a key is found the key is translated to a virtual key and sent to the emulator via the OS. GetResult just moves the content of the specified path to the appropriate folder. The commands to set different values (such as StdKeyPressed) are never added to the command list. They are just affecting the the values of the added commands.

6.12.3 The Virtual Keys

The Windows operating system allows processes to send messages to each other through the OS. This is how TANT sends the virtual keystrokes to the emulator. However TANT is mostly written in Java and Java can not access operating system specific features directly. This have to be done using native code written in C/C++. Java can access code written in C/C++ through an interface called Java Native Interface (JNI) . Through the JNI, TANT can send a virtual keystrokes to the emulator as seen in figure 7.

6.13 Executing a Test Case on Device

The following chapter will discuss how TANT executes test cases on a mobile platform and give a short introduction to AT-commands and the OBEX protocol.

6.13.1 Serial Connection

To communicate and simulate keystrokes on mobile units TANT uses a serial port. Through this port TANT sends both AT-commands and OBEX.

24

Figure 7: Using JNI TANT is able to send virtual keystrokes to the OS

6.13.2 AT-commands

AT-commands dates back to the first days of dial-up modems. They were invented by Dennis Hayes in an attempt to make modems controllable without adding extra pins to the communication port. The original set of AT-commands was very small but have evolved during the years. Most modern cell phones have a built in modem that can understand not just only the basic set of AT-commands, but also some extended command sets and some manufacturer specific once. One of these extended commands are AT+CKPD that can be used to simulate a key being pressed. Using this command TANT can control an execution of a Java MIDlet on a mobile unit. (Wikimedia Foundation 2007a)

6.13.3 OBEX

OBEX is a protocol for object transfers developed by Infrared Data Association®. It allows data to be transferred between two units as binary objects. A typical OBEX session is started by the client sending a connect request to the server. The request contains information such as how big packets the client can handle and what type of service it wants to connect to. If the connection is accepted the server responds with a packet with information on how big packets it can handle and the connection identification number of the current session. When a connection have been established the client can start to send files to, and request files from, the server. A typical OBEX get session can be seen in figure 9.

25

Figure 8: Both AT-commands and OBEX goes through the same serial port

6.13.4 How the Playback is done

When the command list have been generated by the MaL-code parser TANT itterats over it. If a key is found it is translated to the appropriate AT-command using a mobile unit playback configuration file and sent to the mobile unit via the serial port. If an UploadProgram command is found, TANT uploads the MIDlet to the mobile unit using OBEX. OBEX is also used to download the results if an GetResult command is found. Delays are implemented by letting the main thread sleep for the given time.

6.14 Testing Without AT-Commands

Since not all mobile units supports the AT-commands that TANT needs, an alternative have to be presented. The alternative called TestMIDlet will be presented in the following chapter. A short introduction to the MAF will also be presented.

6.14.1 TestEngine

There are many different cell phones on the market and not all of them support AT-commands. One way to control the execution of a MIDlets on such units are to create a subclass to the MIDlet class and let MIDlets inherit from this class instead of the standard MIDlet class. The TestMIDlet is responsible for stepping through the the execution of the MIDlet extending it.

26

Figure 9: Outline of the OBEX communication between a client and a server.

6.14.2 MAF

The MAF is a framework used to write MIDlets that look and act the same on all J2ME implementations. To do this auSystems designed there own graphics and concurrent control classes based on the original J2ME classes.

One of the fundamental classes in MAF is the Engine class. This class implements the J2ME interface Runnable and is a thread safe state engine. This means that when ever a state change occurs, it might be the user selecting an option in a menu, the engine executes the code associated with that state. Once the execution of the code is done the engine sleeps until the next state change occurs. If a state change occurs while the engine is handling a state, the request is blocked until the engine is done.

6.14.3 The TestMIDlet

Most of auSystems MIDlets are based on the MAF. With a few changes to the Engine class a TestMIDlet can be created that can control the state switches of the engine class. Instead of letting the engine start and take control of the state changes, the TestMIDlet dictates to the engine when it can change states. This allows the TestMIDlet to step through an execution of a program. An example can be found in figure 10.

6.14.4 How the Execution is done

All MIDlets must extend the super class MIDlet, and hence must implement the abstract functions startApp, pauseApp, and destroyApp. In MIDlets extending TestMIDlet an extra abstract function called run must be implemented

In a typical MAF-MIDlet-StartApp-function an engine is created and started. In a TestMIDlet the StartApp function instead calls the run function. In the run function the

27

Figure 10: myMIDlet can extend either TestMIDlet for testing purpose or MIDlet for normal execution

execution of the MIDlet is described as calls to static functions imported from a library. These functions are used to evaluate values, change state of the engine and many other things. This library will start as quite small but will continue to grow bigger and bigger for every written TestMIDlet since most of the functions can be reused.

6.15 Result Handling

Results are downloaded and stored in a sub directory to result when a getResult command is executed. All errors and commands of an execution can be viewed when the execution is done. If a database is available, TANT stores the results of the tests, the errors and the commands to the database.

6.15.1 The Database Result Handler

The database result handler handles and presents results, errors and commands of an execution by adding them to a database. Once the executions of a script is done the results, errors and tests are fetched from the database and displayed by TANT.

All the communication with the database is handled by a database manager. The database manager have methods for storing and retrieving data from the database.

6.15.2 The Temporary Result Handler

When no database is available TANT uses a temporary result handler. It creates a temporary directory and stores the downloaded files there. All the results of an executed script are stored in the primary memory of the computer but the user will see no difference between the temporary result handler and the database result handler other than that the results, errors and command logs will not be stored in the database.

At startup all temporary result directories are deleted by TANT.

6.15.3 Screen Shots

A MIDlet executed on two different J2ME implementations may look different. To locate thees differences the execution must be overseen by someone manually. Another solution to this problem is to take screen shots of the screen and evaluate them after the execution. To make capturing of screens possible a screen capturing feature was added to the MAF.

To display images on a screen, MAF draws everything on a Java component called a canvas. The canvas is then interpreted by the runtime environment and displayed on the screen. A canvas can be translated to a Red-Green-Blue (RGB) representation that can be converted to a bit map (BMP) and saved to a file that can be viewed on a computer as seen in figure 11.

28

6.15.4 Execution Loggings

TANT logs all execution errors that it finds to a local file and to a database if one is provided. Some examples of the errors might be that the connection to the cell phone is lost, the emulator could not be started or a syntax error was found in the MaL-code script. The error logging to a file can be switched off.

29

Figure 11: To capture what is on the screen the canvas is turned into a RGB vector and converted into a BMP

7 Discussion and Conclusion

This chapter will discuss the results of the evaluation of the different testing methods presented in this master thesis as well as a concluding discussion.

7.1 Result

The results of the the evaluation of TANT, described in chapter 3.3, can be seen in table 1 below. All three kinds of testing methods succeeded in generating the right results. The manual testing took the least time even excluded the unsupervised time but if the tests were to be repeated the automated tests would be faster since the preparations from the previous tests runs could be reused. Allot of the time in the unattended execution of TANT is due to the delays used when uploading the program.

• Manual Testing: All the time was attended execution time. This include uploading the MIDlet, preforming the tests and verifying the results.

• TANT: The preparation time was spent writing the test scripts and adding log commands to the source code. The attended time was spent verifying the screen shots and test results. The unattended time was when TANT executed the test scripts.

• TestMIDlet: The preparation time was spent writing the test scripts. The attended execution time is the time spent uploading the MIDlet and downloading and verifying the results. The time for the actual execution was so small that we approximated it to 10 seconds. We did this since the time for execution on an emulator was less then 1 second.

Table 1: Results of the evaluation

Method Preparation Time

(mm:ss)

Attended ExecutionTime(mm:ss)

Unattended Execution Time(mm:ss)

All tests yielded correct results?

(mm:ss)Manual Testing

00:00 11:30 00:00 Yes

TANT 15:00 01:00 07:00 Yes

TestMIDlet 10:00 03:00 00:10 Yes

30

7.1.1 Validity

The times measured are not exact. The preparation time of TANT testing may wary very much dependent on the script writers knowledge of the MaL-code and general domain knowledge. The execution time may also wary depending on the hardware used in a test setup.

No usability tests have been preformed so we can not guarantee that the framework is easy to use. However the framework is supposed to be used by people that are interested in programing and are used to computers and software, so we think that it will be easy enough to use.

Due to limited resources the tests were only preformed once for each method. All tests were preformed by the same person. That person was not involved in writing the software that was tested or the test specifications, but was involved in the development of TANT and the TestMIDlet. The tests were preformed on a limited number of devices.

7.1.2 Result Discussion

It is uncertain how the time to write a test script correlates to the complexity and size of the test that the script should execute. Since we only had a simple MIDlet to test on we could not investigate this any further.

If many different test were to be performed there would be more opportunities for reuse of code from the scripts. In our evaluation we only wrote one script for TANT which with some slight modifications were used for all three test cases.

A script can be written based on the test specification simultaneous with the developments of the actual software. This means that the preparation time dose not have to be applied to the testing time of the software at all.

7.1.3 Will TANT be Used?

MIDlets are by nature small and often very graphical intense. Graphics are difficult to test automatically, and even if the tools make it easier it often still would require manual inspection. The investment would often be to large in respect to the gain over manual testing. Also to be considers are the overhead of maintaining the testing tools in a rapidly changing and evolving industry, such as the mobile communications industry. Even now the use of AT-commands are being discontinued along with the phone models that were used for the studies. Therefor we must conclude that the market for TANT as it stands now is very narrow. The writers have after two years working in the mobile communication business, including MIDlet development, concluded that the current implementation for TANT will, most likely, never be used. However through increased awareness of the problems during testing the manual process could be made more effective. In the future when devices have become more powerful and more open for communication, other implementation options might make the task of automating some of the steps required for testing MIDlets on the device more feasible.

31

7.2 Limitations

One of the goals for TANT were to facilitate testing on all platforms. However the TANT framework only supports OBEX and AT through serial connection. Something that works on most phones, but not all. Most notably here is the range of phones from Nokia where a serial connection can not be established. Further more it should be considered as a limitation that OBEX must be supported on the phone however for practical reasons this is not so much of a hinderence as most phones modern enough to support Java also support OBEX. When it comes to support for AT-Commands it is enough for the phone to support some AT-Commands for key strokes since TANT uses a configuration file to determine what AT-Command to send.

For serial connectivity the TANT framework have relied upon a package called RXTX. This however have lead to the occasionally inability to close a port after a failed connection attempt which in turn can mean that the TANT framework will freeze following improper use.

The MIDlet is installed through the use of a .jad file. The .jad file is uploaded to the phone using OBEX. Then the .jar file is downloaded through WAP. This introduced additional limitations considering the connectivity that must be available. However on some phone modules it might be possible under certain circumstances to upload the .jar file directly using OBEX.

7.3 Future Improvements

Given that it is hard to reach perfection in such a limited time span as 20 weeks. There are therefore room for improvements in the future. Some of them we can foresee already and they are therefore discussed here.

With the continued use of the TANT framework there would be a lot of written test cases and test scripts. As of now there aren't any mechanisms for administrating those. It would be more efficient if those could be forged together to form a library making reuse possible. One possible solution could be to integrate a function similarly to a CVS/SVN with TANT.

When we could not find a suitable free implementation of the OBEX protocol, we decided to implement that internally. One possibility would be to turn it into an open source project. Thus easing the burden of maintenance and future updates.

Although TANT incorporates support for the use of a database for logging results from the tests performed. The capabilities for reviewing that data later is lacking. Therefore we would suggest the implementation of a web based interface for viewing the collected data.

As described earlier in chapter 3.2 we use a library called RXTX to communicate over the serial port. Although we feel that this is sufficient, there are some shortcomings to this. When, and if, there are better alternatives made available one should consider changing from the RXTX library.

32

7.4 Conclusion

All in all the results of the thesis lived up to our expectations. We managed to create a framework that is both flexible and easy to use. We found two ways of automatically control the execution of a MIDlet and managed to implement an OBEX interface to send and get files from a mobile platform. We created our own script language and intepratator for it. We also made the MaL-code scripts runnable on both emulator and device without any modification, including uploading the MIDlet to the device and starting the emulator and getting the resulting files from the device or the local file system. If you move the script to another computer nothing should have to be changed. Once a configuration file for a device or emulator have been produced it can be used by every TANT installation everywhere.

However, due to the many variations in different platforms and emulators and the difficulties in getting verifications from the device that a command have been executed, we think that the best results will be achieved if the TestMIDlet method is combined with the TANT framework.

As seen in the evaluation of the testing methods in chapter 8.1 the preparation phase was the most time consuming part of the automated testing. But the execution time is shorter for the automated methods then for the manual testing. The potential gain of the automated testing lye in the repeating of the already written tests.

33

8 Bibliography

Bach, James (1999), Test Automation Snake Oil v2.1.http://www.satisfice.com/articles/test_automation_snake_oil.pdf

Buwalda, Hans – Janssen, Dennis- Pinkster, Iris (2002), Integrated Test Design and Automation Using the TestFrame Method. Pearson Education, Edinburgh

Dustin, Elfriede (1999), Lessons in test automation A manager's guide to avoiding pitfalls when automating testing. Software testing & Quality Engineering, http://www.stqemagazine.com

Ericson, Thomas – Subotic, Anders – Ursing, Stig (1997), TIM a test improvement Model. John Wiley & Sons ltd, http://www.interscience.wiley.com

Kaner, Cam (1997), Improving the maintainability of automated test suites. Software QA, Volume 4, #4, San Francisco.

Karlström, Daniel – Runeson, Per – Nordén Sara (2005), Minimal test practice framework for emerging software organizations. John Wiley & Sons ltd, www.interscience.wiley.com

Krause, Mitchel H (1994), SOFTWARE A Maturity Model for Automated Software tetsing. Medical Device Link, http://www.devicelink.com/mddi/

Linz, Tilo – Daigl, Matthias (1997), How to Automate Testing of graphical User Interfaces. http://www.imbus.de/forschung/pie24306/gui/aquis-full_paper-1.3.shtml

Pettichord, Bret (2001), Seven Steps to Test Automation Success. STAR West, http://www.pettichord.com

Pol, Martin – Teunissen, Ruud – van Veenendaal, Erik (2002), Software Testing A guide to the TMap Approach. Biddels LTD, London.

Sommerville, Ian (2007), Software Engineering 8. Pearson Education, Edinburgh

Wikimedia Foundation, Inc. (2007a), http://en.wikipedia.org/wiki/AT_Commands 2007-01-25

Wikimedia Foundation, Inc. (2007b), http://en.wikipedia.org/wiki/OBEX 2007-01-26

Zambelich, Keith, Totally Data-Driven Automated Testing.http://www.sqa-test.com/w_paper1.html

34

Appendix

35

A Layout of TANT

36

B Test Case Recorder Configuration File for Sony-Ericsson emulator

exeFlag=-Xdescriptor:VK_LEFT=LEFTVK_UP=UPVK_RIGHT=RIGHTVK_DOWN=DOWNVK_NUMPAD0=NUMBER_KEY_0VK_NUMPAD1=NUMBER_KEY_1VK_NUMPAD2=NUMBER_KEY_2VK_NUMPAD3=NUMBER_KEY_3VK_NUMPAD4=NUMBER_KEY_4VK_NUMPAD5=NUMBER_KEY_5VK_NUMPAD6=NUMBER_KEY_6VK_NUMPAD7=NUMBER_KEY_7VK_NUMPAD8=NUMBER_KEY_8VK_NUMPAD9=NUMBER_KEY_9VK_F1=LEFT_SOFT_KEYVK_F2=MIDDLE_SOFT_KEYVK_F3=RIGHT_SOFT_KEYVK_RETURN=ENTERVK_END=MENUE_RETURN_BUTTONVK_F6=INCOMING_CALLVK_F7=END_CALLVK_MULTIPLY=STARVK_SUBTRACT=SQUAREVK_CLEAR=C_BUTTONVK_ESCAPE=POWER

37

C Playback Configuration File for Sony-Ericsson K750 emulatorLEFT_SOFT_KEY=VK_F1RIGHT_SOFT_KEY=VK_F2MIDDLE_SOFT_KEY=VK_F3MENUE_RETURN_BUTTON=VK_ENDC_BUTTON=VK_CLEARUP=VK_UPDOWN=VK_DOWNLEFT=VK_LEFTRIGHT=VK_RIGHTENTER=VK_RETURNNUMBER_KEY_1=VK_NUMPAD1NUMBER_KEY_2=VK_NUMPAD2NUMBER_KEY_3=VK_NUMPAD3NUMBER_KEY_4=VK_NUMPAD4NUMBER_KEY_5=VK_NUMPAD5NUMBER_KEY_6=VK_NUMPAD6NUMBER_KEY_7=VK_NUMPAD7NUMBER_KEY_8=VK_NUMPAD8NUMBER_KEY_9=VK_NUMPAD9NUMBER_KEY_0=VK_NUMPAD0STAR=VK_MULTIPLYSQUARE=VK_SUBTRACTPOWER=VK_ESCAPEINCOMING_CALL=VK_F6END_CALL=VK_F7UPLOADPROGRAM=UPLOADPROGRAMRESTARTPHONE=RESTARTPHONESETUP=SETUP

38

D The Mobile Unit Configuration FileLEFT_SOFT_KEY=AT+CKPD="["RIGHT_SOFT_KEY=AT+CKPD="]"MIDDLE_SOFT_KEY=AT+CKPD="]"MENU_RETURN_BUTTON=AT+CKPD=":R"C_BUTTON=AT+CKPD="C"UP=AT+CKPD="^"DOWN=AT+CKPD="v" LEFT=AT+CKPD="<" RIGHT=AT+CKPD=">"ENTER=AT+CKPD=":J"NUMBER_KEY_1=AT+CKPD="1"NUMBER_KEY_2=AT+CKPD="2"NUMBER_KEY_3=AT+CKPD="3"NUMBER_KEY_4=AT+CKPD="4"NUMBER_KEY_5=AT+CKPD="5"NUMBER_KEY_6=AT+CKPD="6"NUMBER_KEY_7=AT+CKPD="7"NUMBER_KEY_8=AT+CKPD="8"NUMBER_KEY_9=AT+CKPD="9"NUMBER_KEY_0=AT+CKPD="0"STAR=AT+CKPD="*"SQUARE=AT+CKPD="#"POWER=AT+CKPD="p"INCOMING_CALL=ATEND_CALL=ATIDENTIFICATION=ATISETUP=AT+CSCS="8859-1"OBEX=AT*EOBEX

39

E Test Specification

Revision history

Revision Date Name Comment1.0 2007-03-01 Manfred Nilsson Document created.

Introduction

This document describes all the test cases for the product DemoMIDlet. The purpose of this Test Specification is to provide test cases that prove that all input requirements are implemented as specified.

Test object identification

Sub-system / Component Name TANT-Test MIDletSub-system / Component Version

Requirements Baseline

Test case summary

TC ID TC Slogan001 Verify main menu

002 Verify the About screen

003 Verify View screen

40

Test case 001 – Menu choice verification

Information

Description of Test Case Verify that the menu choice corresponds to the right Requirements Covered by Test Case

Test Case Preconditions

Step Description VerificationPre 1 Have a phone capable of running MIDlets. Phone available.

Test Case Set-up

Step Description VerificationSet-up 1 Upload the MIDlet to the phone MIDlet preset on phoneSet-up 2 Install the MIDlet onto the phone MIDlet runnable from phone

Test Case Execution

Step Description Expected OutcomeStep 1 Execute MIDlet MIDlet startsStep 2 Choose menu option View Highlighted value = 0Step 3 Choose menu option About Highlighted value = 1

Test Case Clean-up

Step Description VerificationClean-up 1 Select Menu choice Exit MIDlet shuts down

41

Test case 002 – Verify the About screen

Information

Description of Test Case Bring up the about screen and verify that it is readable and displays in a correct manner.

Requirements Covered by Test Case

Test Case Preconditions

Step Description VerificationPre 1 Have a phone capable of running MIDlets. Phone available.

Test Case Set-up

Step Description VerificationSet-up 1 Upload the MIDlet to the phone MIDlet preset on phoneSet-up 2 Install the MIDlet onto the phone MIDlet runnable from phone

Test Case Execution

Step Description Expected OutcomeStep 1 Execute MIDlet MIDlet startsStep 2 Select menu option About About Menu displayed in a correct

manner.

Test Case Clean-up

Step Description VerificationClean-up 1 Select Menu choice Back Enters main menuClean-up 2 Select Menu choice Exit MIDlet shuts down

42

Test case 003 – Verify the View screen

Information

Description of Test Case Bring up the view screen and verify that it is readable and displays in a correct manner.

Requirements Covered by Test Case

Test Case Preconditions

Step Description VerificationPre 1 Have a phone capable of running MIDlets. Phone available.

Test Case Set-up

Step Description VerificationSet-up 1 Upload the MIDlet to the phone MIDlet preset on phoneSet-up 2 Install the MIDlet onto the phone MIDlet runnable from phone

Test Case Execution

Step Description Expected OutcomeStep 1 Execute MIDlet MIDlet startsStep 2 Select menu option View Picture is Displayed correctly

Test Case Clean-up

Step Description VerificationClean-up 1 Press “Enter” Enters main menuClean-up 2 Select Menu choice Exit MIDlet shuts down

43

F TANT

Settings & Add Emulator

If one dares to enter those dialogs there are a number of settings that can make your life easier if you know what to do with them.

44

A view of the main GUI for the TANT framework.

A view of the setting dialog for TANT

Connection Settings

The TANT framework uses a serial connection to communicate with the phone parameters for this connection is Baud Rate, Flow Control In, Flow Control Out , Data Bits, Stop Bits and Parity.

DB Settings

The TANT frameworks supports the use of a database for storing the results from the tests performed. The option for using the database (connecting or not) can be enable and disabled under the log settings (see below).

In current version only MySQL are supported therefore it is not possible to edit the DB driver setting, it is fixed to “mysql”. The next setting is the DB host constitutes URL for the database. DB username and DB password should be for a database user granted both read and write access for the DB database.

Project Settings

For logging purposes there are the option to enter a name under Your name and for what project you are working on under Project name.

Log Settings

Log file path a complete URL pointing to a file were the logs will be saved. This can be enabled/disabled using the next setting Log to file. Similar the option for logging to a database can be enabled or disabled using the Save results to a database setting. The last setting Enable popups gives the possibility to suppress popup dialogs. This is recommended for unsupervised executing of the tests.

Recording

Recording configuration file ... a file dialog for setting the configuration file which handles the matching between keys in the MaL-code script and VIRTUAL KEY sent to the operating system.

It is possible, even if recommended against, to also log the time when recording a MaL-code script to do so check the setting Advanced Recordning.

45

Add Emulator

The first setting Name is used as a identifier and has to be unique. Next there is the possibility to enter additional flags for starting the Emulator under the setting Executable Flags. Next setting is Destination this is the path to the directory were the emulators “memory” resides.

In order for the TANT framework to send keystrokes to the emulator the TANT framework needs to know which window is the emulator. If Window heading is set to automatic it assumes that the next window to appear after starting the emulator is the emulator. This can however be a bit tricky sometimes with other popups interfering making it unreliable. If the heading is known for the emulator before hand it can be enter or a part of the heading can be entered. Also for more advanced evaluation a Regular Expression can be entered using the prefix RegExp. Also the % can be used as prefix to indicate that the headings ends with the given wording or as a postfix to indicate that it begins with the given wording.

46

Illustration 1: A view of the AddEmulator dialog for TANT