Taming the Testing Beast

Taming the Testing Beast

Marc J. BalcerChief ArchitectModel Compilers LLC

How do you knowwhat to test?

Test Case

TAME – Test Authoring Made Easy

Test Workshee

t TAME

Test CaseTest

CaseTest CaseTest

Case

Excel Open Office

Test Case

Employee

Add an expense to an expense report

Employee

Add an expense to an expense report

Test Case

Expense TypeExpense DateAmountNotes

InputsExpense AddedNo Date ErrorBad Date ErrorAmount Missing ErrorAmount Too Large Warning

Expected Results

“Stateless”

Test Case

Expense TypeExpense DateAmountNotes

InputsExpense AddedNo Date ErrorBad Date ErrorAmount Missing ErrorAmount Too Large WarningNo Project ErrorEmployee Error

Expected Results

Project ExistsEmployee CurrentOther Items Exist

Preconditions

“Stateful”

Test Case

TAME – Test Authoring Made Easy

Test Workshee

t TAME

Test CaseTest

CaseTest CaseTest

CaseInputsPreconditionsExpected Results

Test Suite

How do you geta good set oftest cases?

Example: A Login Page

Inputs

Generic Login Command

Input User ID FieldEmptyToo longHas invalid charactersValid Syntax

Input Password FieldEmptyToo longValid Syntax

Inputs CharacteristicValues

(“choices”)

Kinds of Input ValuesNumeric Symbolic Enumerated Composite

A number

Specified as• Range• Precision

(number of decimal places)

May include a unit of measure

A character string

Specified as• Size• Pattern

(regular expression or equivalent)

A list of values

Specified as the set of values

May be ordered or unordered

May have a corresponding numeric or symbolic value

A single value composed of other values

Examples:• Mailing

Address• Grid

Coordinate

Numeric Type ExamplesType Name Description ExamplesYear A four-digit integer between

1900 and one plus the current year

2012 is good1832 is not good342.34 is not good“huh” is not good

Money A US Dollar value between 0 and 1000.00 to a precision of 2 decimal places.

345.23 is good-60 is not good34.219 is not good

Count A nonnegative number used to indicate a count of items between 0 and 100.

46 is good245 is not good-10 is not good

Date A calendar date to the precision of one day

15-Jan-2010 is good4/3/2010 is good*June 6 is not good

* How this is interpreted depends upon whether you are in the US or Europe…

Symbolic Type ExamplesType Name Description ExamplesISBN Book Number

10 or 13 decimal digitsThe last character may be an X instead of a zero.

0-201-74804-5 is good4-87197-308-5 is good9781492217947 is good9321G4E3 is not good

ISO Country Code

2 capital letters US is goodGB is goodEUR is not good

Order Number 4 digits plus a dash plus 6 digits:[0-9]{4}\-[0-9]{6}(or \d{4}-\d{6})

2014-002432 is good1986-231321 is good503-3324-324 is not good

Title String 1 to 100 characters, including non-ASCII (foreign language) characters

Pretty much anything as long as it’s not more that 100 characters long.

Common Input Values• Errors

• Missing (empty, null, nothing, unchecked, …)

• Too short• Too long• Wrong characters (bad syntax)• Out of range

• Acceptable Values• Good syntax• Special characters

(e.g. quotes, < > characters)• Non-ASCII and non-Latin characters

Generate All Combinations

Results


Error message "Empty User ID" is displayed

User ID Too Long

User ID has invalid characters

Empty Password

Password Too Long

User ID or password incorrect

User ID or password incorrect

User logged in

Input User ID Field

Empty xToo long xHas invalid characters xValid Syntax x x x

Input Password Field

Empty xToo long xValid Syntax x x x

Preconditions




Condition User

no user with the user IDuser with the user ID

Condition User Password

does not match the user IDmatches the user ID

Facts aboutthe environment(usually set up in advance)

Forming Combinations




Condition User

no user with the user IDuser with the user ID


does not match the user IDmatches the user ID

4 choices

3 choices

2 choices2 choices

4 × 3 × 2 × 2 = 48 test cases

All Combinations = Many Useless Test CasesRedundant (duplicate)

User ID emptyPassword too longUser: none with the user IDPassword: does not match the user ID Result: Empty User ID errorUser ID: emptyPassword: good syntaxUser: none with the user IDPassword: does not match the user ID Result: Empty User ID error

Make No Sense

User ID emptyPassword too longUser: user with the user IDPassword: does not match the user IDUser ID bad syntaxPassword too longUser: user with the user IDPassword: does not match the user ID

Define Results


Empty User ID error

User ID Too Long error

User ID has invalid characters error

Empty Password error

Password Too Long error

User ID or password incorrect error


User logged in

Input User ID Field

Empty Too long Has invalid characters Valid Syntax


Empty Too long Valid Syntax

Condition User

no user with the user ID user with the user ID


does not match the user ID matches the user ID

Expected Results

Simple Result Marking


Empty User ID error







User logged in

Input User ID Field

Empty xToo long xHas invalid characters xValid Syntax


Empty xToo long xValid Syntax

Condition User




Combination Results


Empty User ID error







User logged in

Input User ID Field

Empty Too long Has invalid characters Valid Syntax x


Empty Too long Valid Syntax x

Condition User

no user with the user ID user with the user ID x


does not match the user ID matches the user ID x

Multiple Paths to the Same Result


Empty User ID error







User logged in

Input User ID Field

Empty Too long Has invalid characters Valid Syntax x x


Empty Too long Valid Syntax x x

Condition User

no user with the user ID xuser with the user ID x


does not match the user ID xmatches the user ID

same

Fully Marked Worksheet


Empty User ID error







User logged in

Input User ID Field

Empty xToo long xHas invalid characters xValid Syntax x x x


Empty xToo long xValid Syntax x x x

Condition User

no user with the user ID xuser with the user ID x x


does not match the user ID xmatches the user ID x

Forming Test Cases


Empty User ID error





Input User ID Field




Condition User




Test Case 1Inputs: User ID Field : Empty Expected Results: Empty User ID error------------------------------------Test Case 2Inputs: User ID Field : Too long Expected Results: User ID Too Long error------------------------------------Test Case 3Inputs: User ID Field : Has invalid characters Expected Results: User ID has invalid characters error

No Mark – Don’t Care


Empty User ID error





Input User ID Field




Condition User




Test Case 1Inputs: User ID Field : Empty Expected Results: Empty User ID error

Any choice for passwordwill result in this sameerror message.

Default Choice

Select a default if you want to put that “don’t care” category into the test case.

Combine All Choices

Select all choices if you want to combine all choices into test cases.

Alternatively check the category name.

Multiple Paths to the Same Result




Input User ID Field

Empty Too long Has invalid characters Valid Syntax x x


Empty Too long Valid Syntax x x

Condition User

no user with the user ID xuser with the user ID x


does not match the user ID xmatches the user ID

Test Case 6 Preconditions: User : no user with the user ID Inputs: User ID Field : Valid Syntax Password Field : Valid Syntax Expected Results: User ID or password incorrect-----------------------------------------------Test Case 7 Preconditions: User : user with the user ID User Password : does not match the user ID Inputs: User ID Field : Valid Syntax Password Field : Valid Syntax Expected Results: User ID or password incorrect

Combination Results


User logged in

Input User ID Field

Empty Too long Has invalid characters Valid Syntax x


Empty Too long Valid Syntax x

Condition User




Test Case 8 Preconditions: User : user with the user ID User Password : matches the user ID Inputs: User ID Field : Valid Syntax Password Field : Valid Syntax Expected Results: User logged in

Multiple CombinationsGeneric Login Command (With Special

Inputs)

User logged in

Input User ID FieldEmptyToo longHas invalid charactersValid Syntax xValid with non-ASCII chars x

Input Password FieldEmptyToo longValid Syntax xValid with quotes xValid with HTML chars xValid with non-ASCII chars x

Condition Userno user with the user IDuser with the user ID x

Condition User Passworddoes not match the user IDmatches the user ID x

Form all combinationsof these inputs.

2 × 4= 8 test cases

LimitsGeneric Login Command (With Special

Inputs)LIMITS

User logged in

Input User ID Field

Empty Too long Has invalid characters Valid Syntax xValid with non-ASCII chars 1 x


Empty Too long Valid Syntax xValid with quotes 1 xValid with HTML chars 1 xValid with non-ASCII chars 1 x

Condition User




Each choice will only beused in one test case.

Generating Test Cases

Test CaseTest

Worksheet TAME

Test CaseTest

CaseTest CaseTest

Case

Excel Open Office

Test Case List

Test SummaryTable

Test ProtocolDocument

Test Case ListTest Cases for Generic Login Command (With Special Inputs)----------------------------------------------------Test Case 1 Preconditions: Inputs: User ID Field : Empty Expected Results: Empty User ID error----------------------------------------------------Test Case 2 Preconditions: Inputs: User ID Field : Too long Expected Results: User ID Too Long error----------------------------------------------------Test Case 3 Preconditions: Inputs: User ID Field : Has invalid characters Expected Results: User ID has invalid characters error

Test Summary TableUser ID Field Password Field User User Password Results...

1Empty Empty User ID error2Too long User ID Too Long error

3Has invalid characters


4 Empty Empty Password error5 Too long Password Too Long error6Valid Syntax Valid Syntax no user with the user ID User ID or password incorrect

error

7Valid Syntax Valid with quotes no user with the user ID User ID or password incorrect error

8Valid Syntax Valid with HTML charsno user with the user ID User ID or password incorrect error

9Valid Syntax Valid Syntax user with the user ID does not match the user ID User ID or password incorrect error

10Valid Syntax Valid with quotes user with the user ID does not match the user ID User ID or password incorrect error

11Valid Syntax Valid with HTML charsuser with the user ID does not match the user ID User ID or password incorrect error

12Valid Syntax Valid Syntax user with the user ID matches the user ID User logged in

13Valid Syntax Valid with quotes user with the user ID matches the user ID User logged in

14Valid Syntax Valid with HTML charsuser with the user ID matches the user ID User logged in

Default Protocol Document

Creating a Custom ProtocolHow to log in:• Click a link to go to the login page• Enter the user ID• Go to the password field• Enter the password• Click the login button• Observe whether

• an error message is displayed, or• the application indicates that the user is

logged in

Creating a Custom ProtocolHow to log in:• Click a link to go to the login page• Enter the user ID• Go to the password field• Enter the password• Click the login button• Observe whether

• an error message is displayed, or• the application indicates that the user is

logged in

We want real valuesfor each of these parameters.

Defining Values

Generic Login Command (With Special Inputs)

LIMITS

Empty User ID error







User logged in

User ID:

User ID: abcdefghijklm

User ID: 56%#@!*(a

User ID: gary*

User ID: charlie

User ID: garcía

Password:

Password: 56%#@!*(a

Password: Good*

Password: Bad

Password: G"oo"d

Password:Go<o>d&

Password:¿sueÑo

Input User ID Field

Empty x xToo long x xHas invalid characters x xValid Syntax x x x x xValid with non-ASCII chars 1 x x


Empty x xToo long x xValid Syntax x x x x xValid with quotes 1 x x x xValid with HTML chars 1 x x x xValid with non-ASCII chars 1 x x

Condition User

no user with the user ID x xuser with the user ID x x x x x x x


does not match the user ID x xmatches the user ID x x x

Defining Values

User ID:


User ID: 56%#@!*(a

User ID: gary*

User ID: charlie

User ID: garcía

Password:

Password: 56%#@!*(a

Password: Good*

Password: Bad

Password: G"oo"d

Password:Go<o>d&

Password:¿sueÑo

Parameter Name : value

Custom Protocol TemplateTest Case «TestCaseNumber» Preconditions

«TableStart:Precondition»«CategoryName»

«ChoiceName»«TableEnd:Precondition»

Inputs «TableStart:Input»«CategoryName»

«ChoiceName»«TableEnd:Input»

Results «TableStart:Result»«ResultName»«TableEnd:Result»

Procedure: (Warning: If you customize this section do not use numbered lists. Word MailMerge doesn’t have the sense to reset the numbering.)

Open the website Click the Login button Enter «User ID» into the User ID field. Hit tab Enter «Password» into the password field Click the Login button

Now verify that: «TableStart:Result»

«ResultName»

«TableEnd:Result»

WordMailmergeFields

User ID and Passwordvalues filled inhere

Simple ValuesGeneric Login Command (With Special

Inputs) User ID:


User ID: 56%#@!*(a

User ID: gary*

User ID: charlie

User ID: garcía

Password:

Password: 56%#@!*(a

Password: Good*

Password: Bad

Password: G"oo"d

Password:Go<o>d&

Password:¿sueÑo

Input User ID Field

Empty xToo long xHas invalid characters xValid Syntax x xValid with non-ASCII chars x


Empty xToo long xValid Syntax x xValid with quotes xValid with HTML chars xValid with non-ASCII chars x

Condition User

no user with the user ID xuser with the user ID x x x x x


does not match the user ID xmatches the user ID x x

Complex ValuesGeneric Login Command (With Special

Inputs) User ID:


User ID: 56%#@!*(a

User ID: gary*

User ID: charlie

User ID: garcía

Password:

Password: 56%#@!*(a

Password: Good*

Password: Bad

Password: G"oo"d

Password:Go<o>d&

Password:¿sueÑo

Input User ID Field




Condition User




Complex ValuesGeneric Login Command (With Special

Inputs) User ID:


User ID: 56%#@!*(a

User ID: gary*

User ID: charlie

User ID: garcía

Password:

Password: 56%#@!*(a

Password: Good*

Password: Bad

Password: G"oo"d

Password:Go<o>d&

Password:¿sueÑo

Input User ID Field




Condition User




Default Values

Generic Login Command (With Special Inputs)

LIMITS

Empty User ID error



User ID: 56%#@!*(a

User ID: gary*

User ID: charlie

User ID: garcía

Password:

Password: 56%#@!*(a

Password: Good*

Password: Bad

Password: G"oo"d

Password:Go<o>d&

Password:¿sueÑo

Input User ID Field

Empty xToo long x xHas invalid characters xValid Syntax x xValid with non-ASCII chars 1 x


Empty xToo long xValid Syntax x xValid with quotes 1 xValid with HTML chars 1 xValid with non-ASCII chars 1 x

Condition User




* indicates thedefault value

Values Applied to the ProtocolTest Case 7 Preconditions

User no user with the user ID

Inputs User ID Field Valid Syntax Password Field Valid with quotes

Results User ID or password incorrect error

Procedure: Open the website Click the Login button Enter charlie into the User ID field. Hit tab Enter G"oo"d into the password field Click the Login button

Now verify that: User ID or password incorrect error

Going Beyond the Basics• Create variability in procedures

• Action Choices

• Using TAME when bugs are found• Root Cause Analysis• Modify the Tests

• Scaling Up• Higher Level Tests• Results become Preconditions

Additional variabilityTest Case 7 Preconditions






- Enter a URL directly- Click a link on the company

homepage- Click a link on an unsecured

page- Tab to the next field- Click into the password

field- Click the Login button- Tab to the Login button

and press Enter

Action ChoicesTest Case 7 Preconditions






• Enter a URL directly• Click a link on the company

homepage• Click a link on an unsecured

page• Tab to the next field• Click into the password

field• Click the Login button• Tab to the Login button

and press Enter

For each of these• Create “action”

choices• Use the action

choices in the protocol

Unforseen Circumstances

A user ID that includes spaces causes a crash

Three failed logins lock the user’s account

• Add “contains spaces” as a choice

• Add a value definition that includes spaces

• Determine how to handle a user ID that includes spaces: is it allowed or is it an error?

• Add “Preceding failed logins” as a new preconditions with several choices

• Determine whether a locked account presents a different error message

• Add result selectors

Scaling Up

Employee hasexpenses to report

Log In

Create NewExpense Report

Add ExpenseItem

Submit toManager

Expense Report Submitted

OpenExpense App

already logged in

not logged in

Login is just one step in a larger process

Scaling UpEmployee hasexpenses to report

Log In


Add ExpenseItem

Submit toManager


OpenExpense App

already logged in

not logged in

Create separatetest specificationsfor each stepin the process

Scaling Up – Business Processes

Employee

Manager

Controller

Employee has expenses to report

RecordExpenses

ApproveExpenseReport

PayReimbursement

Reimbursement

RejectExpenseReport

ResubmitExpenseReport

Submitted Expense Report

Rejected Expense Report

Resubmitted Expense Report

Approved Expense Report

Review New Expense Report

Review Resubmitted Expense Report

CancelExpenseReport

Expense Reporthas been canceled

Consider whether to resubmit

Employee hasexpenses to report

Log In


Add ExpenseItem

Submit toManager


OpenExpense App

already logged in

not logged in

Generating Test Cases

Test CaseTest

Worksheet TAME

Test CaseTest

CaseTest CaseTest

Case

Excel Open Office

Test Case List

Test SummaryTable

Test ProtocolDocument

Comparative Testing Tools• Run Tests

• White-box, code-centricNUnit, JUnit

• UI-based, capture-playbackRanorex, Telerik

• Design by Contract• Write a specification, construct a single

test caseCucumber / Gherkin

Where to use TAME• Unit testing

• specify and generate test cases that can run against software modules (classes, functions).

• GUI testing• whether a traditional installed application, a

website, or a mobile app, use TAME to specify the contents of UI pages, then generate test cases to validate user-facing behavior.

• Regression testing• update the test specification and the test

cases whenever new features are deployed or bugs are fixed.

Where to use TAME• Project Planning

• create initial test specifications to estimate the number of test cases to be run and project the cost of the test effort.

• Retrospectives• use the test specifications to probe

features to better understand the root causes of failures.

TAME for All• TAME helps the analyst

to think through what needs to be tested.

• TAME gives the tester a platform for creating many test cases quickly.

• TAME gives the project manager a means for sizing and scoping the testing effort.

TAME software

Training Courses“Writing Testable

Requirementsand Effective Test Cases”

.

“Modeling Techniquesfor Business Analysts”

Thank you

Marc J. BalcerChief Architect

Model Compilers [email protected]

http://www.modelcompilers.com

Software

Taming the Testing Beast