Projects in Software Testing

Software Testing, ETS 200 Programvarutestning, 5p.

Version 1.0, 2008, Lp Vt2 Per Runeson, Yeni Li Helgesson, CS, LTH

http://www.cs.lth.se/ETS200/ 1. Introduction This document gives the practical details regarding the project in the course Software Testing. The project in the course is equivalent to 2 weeks of full time studies (2 credit points). The main objective of the project is to gain a deeper understanding of a specific area within verification and validation (V&V) of software products. A project group consists of 4-5 persons, who perform the project together. All project members should be involved and the total effort should be evenly distributed among participants. 2. Learning Objectives The objective of the project is to learn a specific area of software testing. Furthermore, the structure of the project makes it possible to learn important parts that will be practiced later on in the education. Hence, the main learning points of the project are: • Learn a specific area of software testing • Collect and summarize research information • Critical thinking beyond the written information • Present information in a structured way There are several types of projects to choose among. Examples of project types are: • Research: solve a research problem; survey the state-of-the-art and identify the research

problems in some area; develop and justify an extension to an existing technique; etc. • Evaluation: apply and evaluate a technique or evaluate a commercial testing or, analysis

tool. • Practical: Use an existing technique to test a system or design and implement a

prototype for a system. 3. Activities

The main activities in the project are:

1. Decide on a subject. There are some subjects specified in this document. Other subjects may be chosen, but has to be decided together with your project supervisor.

2. Find literature. In each subject area, there is a suggestion of research reports to start reading. Other research literature can be found on http://elin.lub.lu.se (within the domain of LU, or by using your STIL password). For research projects, 7-10 reports should be chosen. For evaluation and practical projects, fewer reports are needed.

3. Outline the report. Specify heading one and two and write in short sentences what will be included in the sections.

4. Read; perform evaluation and practical work. Read and understand the new area of software testing; perform evaluation and practical work if these kind of projects have been chosen.

5. Write the report. The report is preferable written in English. The report shall be written in the IEEE template, see the homepage of the course.

6. Present the report. Communicate the area to other people who have knowledge of software testing, but not the specific area. This is done during the presentation session. You have about 15 minutes to describe your work. The presentation is preferable in English.

4. Schedule Week Activities Hand in to supervisor Deadline

14 Decide on a subject Decided subject Thu 3/4 Search literature 15-16 Search literature Read literature Outline the report Outline Tue 15/4

17-18 Read literature Write report Meeting with supervisor Book time 19 Write report Report to supervisor Prepare presentation

Presentation Session Tue 13/5 20

Update report If needed, updated report to supervisor Wed 21/5 21

5. Assessment

See the course program. 6. Report

The report shall be written using the IEEE template. The report shall consist of: • Abstract: summary of all parts in the report. The purpose of the abstract is to attract

people to read your report. • Introduction: introduce the chosen area. The purpose is to give an introduction to the

reader who is not familiar to the specific area, but knows software engineering and testing generally.

• Description of the chosen area: summary of the chosen area. The purpose is to describe why, how and for which purpose the area is used.

• Analysis: analysis, critical thinking and future plans within the chosen area. For research projects, the purpose is to describe the benefits, drawbacks, what research the area needs in the future. For evaluation and practical projects, the work should be described here together with a shorter analysis.

• Conclusion: main conclusion in the report. The purpose is to discuss the main points in the report again. Another structure can be chosen after discussion with your project supervisor.

7. Presentation The project will be presented in the project presentation sessions at the end of the course. Each project will be given 15 minutes to present their chosen area. The presentation should cover the main points of summary, analysis and conclusion.

8. Project areas There are several types of projects to choose among. For research projects, 7-10 reports should be chosen. For evaluation and practical projects, fewer reports are needed. Examples of project types are: • Research: solve a research problem; survey the state-of-the-art and identify the research

problems in some area; develop and justify an extension to an existing technique; etc. • Evaluation: apply and evaluate a technique or evaluate a commercial testing or, analysis

tool. • Practical: Use an existing technique to test a system or design and implement a

prototype for a system. If you would like to choose another area, describe the area to your project supervisor before starting the work. The keywords in Section 9 can be used when you chose an area and when you search for literature. 8.1 How do you know your test cases are correct? Test cases are developed to check whether a software system is implemented correctly according to the requirements specification. However, if the test cases are badly chosen, they will not detect the failures. There is a method, called Mutation testing, which can be used to check whether the test cases are correct. Mutation Testing: A. M. R. Vincenzi, J. C. Maldonado, E. F. Barbosa and M. E. Delamaro, “Unit and Integration Testing Strategies for C Programs using Mutation”, Software Testing, Verification and Reliability, 11(3):249–268, 2001. 8.2 How can you measure that the software system works? Software testing is performed to detect the failures in the software. There are different metrics to use in order to measure whether the software is correct. One such measure is reliability, which is defined as: “The probability for a failure-free operation of a program for a specified

time under a specified set of operating conditions.” (IEEE 610.12-1990). There are several models and techniques that can be used, for example, reliability growth models, Markov models, statistical usage testing, usage-based testing and operational profile testing. Reliability growth models: C. Stringfellow, A. A. Andrews, “An Empirical Method for Selecting Software Reliability Growth Models”, Empirical Software Engineering, 7(4): 319-343, 2002. Markov models: J. A. Whittaker and M. G. Thomason, “A Markov Chain Model for Statistical Software Testing”, IEEE Transactions on Software Engineering, 20(10):812-824, 1994. 8.3 What techniques are most effective to check the requirements and design documents? Static verification is often used in the beginning of the development of software. There are several different techniques and methods used to check the static representation. The com5 mon feature of these is that they have to be manually checked by reviewers. Software inspection, reviews, walkthroughs are common techniques, which are used together with reading techniques. Software inspection and reading techniques: A. Aybuke, H. Petersson, C. Wohlin, “Stateof-the-art: Software Inspections after 25 Years”, Software Testing, Verification and Reliability,12(3):133-154, 2002. 8.4 How can you perform testing on an object-oriented software system? A special branch of software testing is application-based testing. Object-oriented is one such application, which needs special treatment and special methods to test the system. Object-oriented testing: Y. Labiche, P. Thévenod-Fosse, H. Waeselynck, M.-H. Durand, “Testing Levels for Object-Oriented Software”, Proceedings of the 22nd International Conference on Software Engineering, pp. 136-145, 2000. 8.5 How can you perform testing on a distributed system? A special branch of software testing is application-based testing. Testing distributed systems need special treatment and special methods. Testing of distributed systems: S. Goeschl and H. M. Sneed, “Case study of testing a distributed internet-system”, Software Testing, Verification and Reliability, 12(2):77 - 92, 2002. 8.6 What techniques can be used to know when to stop testing? Stopping criteria are used in the testing phase to determine when a certain quality level has been achieved. The quality can for example be defined as the reliability or just the number of faults left in the system. There are several techniques to use as a stopping criterion. One could, for example, estimate the number of faults left after an inspection or to estimate the number failures left in testing. Fault content estimation; L. C. Briand, K. E. Emam, B. G. Freimut, O. Laitenberger, “A Comprehensive Evaluation of Capture-recapture Models for Estimating Software Defect Content”, IEEE Transactions on Software Engineering, 26(6):518-540, 2000. Stopping criterion in testing: J. D. Musa and A. F. Ackerman, “Quantifying Software Validation: When to Stop Testing?”, IEEE Software, 6(3):19-27, 1989. 8.7 How is testing performed for Agile processes? Test-first is a principle that is often used in Agile processes. The principle is based on that test cases are first derived and then these are used to specify how the system should work as well as used as test cases when the system is developed. There are some questions, which are of interest within this area, for example, what is the relation between requirements and test cases, what is the final quality of such a system and how easy is it to maintain such a system? Test-first principle: M. M. Muller, and O. Hagner, “Experiment about Test-first Programming”, IEE Proceedings Software, 149(5):131-136, 2002.

8.8 How can different testing techniques (methods) be compared? One research methodology is to use empirical methods in order to evaluate which testing methods are best to use. The methodologies are often divided into experiments, surveys and cases studies. All these methods are important in order to help software organizations to choose the right testing technique for their purpose. Experiment: Sun Sup So, Sung Deok Cha, Timothy J. Shimeall and Yong Rae Kwon, “An Empirical Evaluation of Six Methods to Detect Faults in Software”, Software Testing, Verification and Reliability, 12(3):155–171, 2002. Survey: P. Runeson, C. Andersson and M. Höst, “Test Processes in Software Product Evolution- A Qualitative Survey on the State of Practice”, Journal of Software Maintenance and Evolution, 15(1):41-59, 2003. Case study: T. Berling and T. Thelin, “An Industrial Case Study of the Verification and Validation Activities”, International Software Metrics Symposium, pp. 226-238, 2003. 8.9 What quality models are used for the testing purpose? There are many general quality models developed within software engineering, and specific models for testing. The quality models try to capture the essential parts of software development (testing) so the final quality will be assured. Some of the general models are CMM, SPICE, ISO 9000, and some of the specific models are TPI and TIM. CMM: M. Paulk, B. Curtis, M.B. Chrissis, C.V. Weber, “Capability Maturity Model, Version 1.1”, IEEE Software, pp. 18-27, 1993. SPICE: J-M. Simon, ‘‘SPICE: Overview for software process improvement’’, Journal of Systems Architecture, 42(8):633-641, 1996. TIM: T. Ericson, A. Subotic, S. Ursing, ‘‘TIM --- A Test Improvement Model’’, Software Testing, Verification and Reliability, 7(4):229-246, 1997. 8.10 How is the software of a nuclear plant tested? Safety is concerned with the property of a system that it will endanger human life or the environment. A safety-critical system is, hence, one by which the safety of equipment or plant is assured. An example of such a system is an aeroplane or nuclear plant system. Safety-critical software needs special attention when it comes to testing. Safety: N. G. Leveson, S. S. Cha and T. J. Shimeall, “Safety Verification of ADA Programs using Software Fault Trees”, IEEE Software, 8(4):48-59, 1991. 8.11 How can test cases be derived and generated? Software test cases are often derived from documents produced during the software development. There are different techniques and languages that can be used. Use cases can be used to derived test cases and two methods that use this approach are usage-based testing and operation profile testing. Furthermore, there are techniques that are automated and use UML diagrams. Usage-based testing: B. Regnell, P. Runeson and C. Wohlin, “Towards Integration of Use Case Modelling and Usage-Based Testing”, Journal of Systems and Software, 50(2):117-130, 2000. Operational profile testing: J. Musa, “Operational Profiles in Software-Reliability Engineering”, IEEE Software, 10:(2):14-32, 1993. Specification-based testing: J. Offutt, S. Liu, A. Abdurazik and P. Ammann, “Generating Test Data from State-based Specifications”, Software Testing, Verification and Reliability, 13(1):25–53, 2003. 8.12 How can process simulation be used for project planning and learning? Software process simulation is used in a variety of fields in software engineering, for example, to support process improvement, software project management training, decision support and to understand empirical research results. This project can either be practical by focusing on developing a simulation model of a software development/testing project with the

pupose to assist in the project planning, or it could be focused on evaluating existing models’ usefulness in the decision-making. Simulation models for decision support: Donzelli, P., “A Decision Support System for Software Project Management”, IEEE Software, 23(4): 67-75, 2006. Why using process simulation models: Kellner, M., Madachy, R., Raffo, D., "Software Process Simulation Modeling: Why? What? How?", Journal of Systems and Software, 46(2- 3):91-105, 1999. 8.13 How can tools for software testing be evaluated? Tools are important in order to implement software testing effectively in an software organization. However, although tools are needed, they do not solve the problems in the testing phase. Several tools exist and they have to be evaluated before a software organization purchases one. Evaluation of testing tools: Poston, R. M. and Sexton, M. P., “Evaluating and Selecting Testing Tools”, IEEE Software, pp. 33-42, 1992. 8.14 How can predictions of the software development project be useful during the testing process? The project management is often interested in predicting the outcome of the ongoing software development projects. Both predictions of how much resources that will be required to deliver on time, and the quality of the delivered software product are valuable for the planning process. What metrics are useful for predictions during the test process, and how can these be used? One model to use for the whole development project is COCOMO, but other approaches could be applied when focusing on the test process. COCOMO and other models: Boehm, B., Abts, C., Chulani, S., “Software Development Cost Estimation Approaches – A Survey”, Annals of Software Engineering, 10(1-4):177-205, 2000. What is needed for making measurements: Schneidewind, N. E., “Body of Knowledge for Software Quality Measurement”, IEEE Computer, 35(2):77-83, 2002. 8.15 How can web applications be tested? The quality of web applications mostly relies on the skill of the individual developer. In order to meet the quality decided by end-users, the processes of developing and testing web applications need to be formalized. Web application testing: Di Lucca G. and Fasolino, A., “Testing Web-based Applications: The State of the Art and Future Trends”, Information and Software Technology, 48(12):1172-1186, 2006. 8.16 How do you effectively regression test your system? Regression testing is the process of validating modified software to detect whether new errors have been introduced into previously tested code. Because of time and resource constraints for testing, regression test selection techniques have been proposed, to reduce the expenses. Regression testing: Rothermel, G. and Harrold, M.J., “Analyzing regression test selection techniques”, IEEE Transaction on Software Engineering, 22(8):529-551, 1996.

9. Keywords in Software Testing The below keywords can be used if you want to decide on a subject not specified above. Another source that is appropriate to use is the Software Engineering Book of Knowledge. The latest version can be downloaded on http://www.swebok.org.

White-box testing Black-box testing Gray-box testing

Equivalence partitioning Boundary value analysis Decision table Finite-state machine-based Testing from formal specification Error guessing Random testing Operational profile SRET

Reference models from code-based testing (flow-graph, call-graph) Control flow-based criteria Data-flow based criteria Mutation testing Coverage measures

Application testing Test Methods Testing in Software Process

Object-oriented testing Component-based Web-based GUI testing Testing of concurrent programs Protocol conformance testing Testing of distributed systems Testing of real-time systems Testing of scientific software

Connection between require-ments and testing Architecture Testability

Statistical usage testing (usagebased) Specification-based software testing Code-based, Fault-based Testing from formal specification Random testing

Testing metrics and measurements Objectives of Testing Evaluation of the testing

Acceptance testing Installation Alpha, beta Conformance, functional, correctness Regression Performance Stress Back-to-back Recovery Configuration Usability

Attitudes and egoless program-ming Test process Test documentation and test-wares Test organisation vs. company Size Cost/effort estimation and other metrics Termination and stop criterion Test reuse and test patterns

Coverage measures Fault seeding Mutation score Comparison and relative effec-tiveness of different techniques Stopping criteria

Empirical methods in software testing

Quality models and software testing Estimations in software testing CMM SPICE Testing with SQA Testing and certification

Reliability Reliability growth models, types and classification of faults Remaining faults and faults density

Compare testing techniques Compare inspection and testing

Testing on the management level Inspections Testing targets

Unit Integration System Alpha Beta

Process Reading techniques

Automated software testing Simulation of the testing proc- Testing connected to

develop (tools) ment process ess

Tools for different techniques XP Cleanroom

Journals • ACM SIGSOFT software engineering notes /Software engineering notes

• International Conference on Software Engineering

• ACM transactions on software engineering and methodology • Automated Software Engineering • Empirical Software Engineering • IEEE Software • Information and Software Technology • Journal of Systems and Software • Software - Concepts & Tools • Software Engineering Journal • IEEE Transactions on Software Engineering • Software Engineering. IEE Proceedings • Software Quality Journal • Software Testing, Verification and Reliability

Conferences

• International Conference & Workshop on the Engineering of Computer-Based Systems • International Symposium on Empirical Software Engineering • International Symposium on Software Metrics • International Conference on Automated Software Engineering • International Symposium on Software Reliability Engineering

Below, there are references to test and process-related conferences. These are almost the same references as in Appendix I in the book by I. Burnstein.

10. Journals and Conferences Below, there are some journals and conferences listed which publish results of software testing research.

11. Test-Related References

Risk-based

Software Testing: Related Conferences Test-Oriented Web Sites •www.sqe.com •www.rbsc.com •www.stickyminds.com •www.softwareqatest.com •www.testinginstitute.com •www.ssq.org •www.stc-online.org •standards.ieee.org •www.qualityweek.com •www.ondaweb.com Software Process and Quality Information •www.mtsu.edu/~storm •www.sei.cmu.edu/collaborating/spins/spins.html

•www.stqemagazine.com

•www.sei.cmu.edu •www.soft.com •www.software.org •www.sast.se •www.espi.org •www.aptest.com •www.sqi.gu.edu.au/spice/contents.html •www.testingeducation.com •www.asq.org •www.opensourcetesting.org

•www.qaiworldwide.com