6656075 Software Testing Chapter10

Test Cases

Writing & Tracking Test Cases

What is a test case?

A set of inputs, execution conditions and expected results developed for a particular objective.

The smallest entity that is always executed as a unit from beginning to end.

Goals of Test Case Planning?

Carefully & Methodically planning test cases is important.

Why?Organization.Repeatability.Tracking.Proof of testing / not testing.

Organization: Even in small project possible to have

thousands of tests cases. Test cases may be created by several

testers over a long span. Proper planning will organize test cases

so that all testers and other project members can review & use them effectively.


Repeatability Necessary to run the same tests

cases several times to look for new bugs.

Without proper planning it would be impossible to know what test cases were last run and how they were run.

Keeping info on this will help to repeat the exact tests.


Tracking We may to answer several questions

like: How many test cases planned for run? How many test cases were run in the

last release? How many passed & how many failed? Any tests cases skipped? Planning would help to answer these

questions easily.


Test Case Planning Overview

TEST PLAN

Test Design Spec

Test DesignSpec

Test CaseSpec

Test ProcedureSpec

Increasing Emphasis onProcess of Planning

Increasing Emphasis onWritten Plan

Test Design Spec

Take the high level overall test plan.Break it into specific features and testable items.Assign the items to individual testers.Only general mention of automation, black-box or white-box.No details.Where to get next level of detail??

Go to test design specification.

ANSI/IEEE 829:Refines the test approach defined in the test plan.Organizes and describes the testing that needs to be performed on a specific feature.Identifies: the features to be covered by the

design and its associated tests. the test cases & test procedures

required to accomplish testing and mentions the pass/fail criteria.

What is Test Design Spec?

Identifiers: A unique identifier used to reference

and locate the test design spec. The spec should reference the overall

test plan. Should contain pointers to any other

plans or specs that it references.

Parts of Test Design Spec

Features to be Tested. Description of s/w feature covered by

the test design. Ex. “The addition function of Calculator”.

Also identify features that may be indirectly tested as a side effect of testing the primary feature. (UI for load/stress)

Also list features being covered in this plan but wont be tested.


Approach Description of the general approach that will

be used to test the features. Expand on the approach to describe the

technique to be used and explain how the results will be verified.

Ex. A testing tool will be developed to sequentially load and save pre-built data files of various sizes. The # of data files, sizes and the data they contain will be determined through black-box tests and supplemented with white-box. A pass or fail will determined by comparing using the file compare tool.


Test Case Identification

A high level description and references to the specific tests cases that will be used to check feature.

List selected equivalence partitions and provide references to the test cases and test procedures used to run them.

Ex. Check the highest possible valueTest Case ID# 15326


Pass/Fail Criteria

Describe exactly what constitutes a pass and a fail of the tested feature.

Must be simple and clear. No doubt on what constitutes a pass

and a fail. Can be fuzzy. (10% or more test

cases fail)


What is Test Case Specification?

How to document the test cases created. ANSI/IEEE 829.A test case specification documents the actual values used for input along with the anticipated outputs.A test case also identifies any constraints on the test procedure resulting from use of the specific test case.

Parts of Test Case Specification

Identifiers A unique identifier for the test case spec to be

referenced by the test design spec and test procedure.

Test item Describes the detailed feature, code module

and others being tested. More detailed description than test design

spec. Ex. Test Design Spec: “The addition function of

Calculator”. Test Case Spec: “Upper limit handling of

addition calculation”.

Input Specification: Lists all I/Ps or conditions given to the

s/w to execute the test case. Ex. To a calculator addition function it

would be 1+1

Output Specification: Describe the result expected from

executing the test case. Ex. Did 1+1 =2?


Environmental Needs: H/w, s/w, test tools, facilities, staff etc

needed to run the test case.

Special Procedural Requirements: Describes anything unusual that must

be done to perform the test. Ex. Testing nuclear power plant need

special requirements.


Test case dependencies: Is a test case dependent on any other

test case? Document that info here.


Test Case Presentation

Can be presented as: Simple Lists. Outlines. Graphical Diagrams: state table/data

flow diagrams Matrix/table form.

Test Case Id: Unique identification number. Simple scheme T-number.

Test Outline: The item in the test case spec from

which the test case derived. Helps to track traceability.

Test Case Presentation: Table Format

Prior State: Idle state Successful execution of another test.

Input: The input sequence to be entered by the

tester.

Expected Results: Behaviour the tester expects to observe.

Actual Results: To be recorded after execution.



TestCase ID

TestOutline

PriorState

Input ExpectedResults

ActualResults

T-1 1.1.1.1 Idle 199 Display:Error

TestCase ID

PrinterMessage

Model Mode Options

T-1 1.1.1.1 Idle 199 Display:Error

A sample test caseIdentificationTest case ID: _____ S/W Version: ______Subsystem: username field test OS: _____Tester’s Name: _____ Date of Test: ____________________________________________________Initial Setup1. Bring up the login dialog box.2. …._______________________________________________Input:1. Place the cursor in the username field2. 2….____________________________________________________Expected Results:……….__________________________________________________Actual Results: Pass Fail

Test ProceduresAfter test design and test cases we need procedures required to execute the test cases.ANSI/IEEE 829.Test procedure specification identifies all steps required to operate the system and exercise the specified test cases in order to implement the associated test design. Also called test script spec.

Parts of Test Procedure Spec

Identifier: Unique identifier to tie the test procedure with

the associated test design and test case.

Purpose: The purpose of the procedure and reference to

the test cases that it will execute.

Special Requirements: Other procedures, special testing

skills/equipment etc needed.

Procedure Steps: Detailed description of how the tests are to run.

Log: Tells how and by what methods the results and

observations will be recorded.

Setup: Explain how to prepare for the test.

Start: Explain how to prepare for the start of the test.

Procedure: Describes the steps used to run the tests.

Measure: Describe how the results are to be determined:

with stop watch/visual examination/…..


Shut Down: Explain steps for suspending the test for

unexpected reasons.

Restart: How to pick up the test at a certain point if there

is a failure.

Stop: Describes the steps for an orderly halt.

Wrap Up: How to restore the environment to its pre-test

condition.

Contingencies: Explains what to do if things don’t go as planned.


Example of a test procedure

Identifier: WinCalcProc98.1872Purpose: This procedure describes the steps necessary to

executeAddition function test cases WinCalc98.051 through

WinCalc98.0185Special Requirements: No special h/w or s/w is required

other thanwhat is outlined.Procedure Steps:

Log: The tester will use WordPad with the Testlog template as the means for taking notes while…..Setup: The tester must install a clean copy of Win98….Start:Procedure:

Organizing & Tracking Test Case

In your head: if for own use.Paper/Documents.Spreadsheet.Custom d/b.

Test Case Management ToolsWhen there is a need to manage several disorganized tests.The best test case managers are: Provide a user interface for managing tests. Organize tests for ease of use and maintenance. Start and manage test execution sessions that run

user-selected tests. Provide seamless integration with capture/playback

and coverage analysis tools. Provide automated test reporting and documentation.

Though several leading vendors claim to accomplish this the bad news is we are still not there.

Write a test design spec, test case spec and test procedure spec.

Ex. Performing the four basic mathematics operations +, -, *, / on the calculator.