Chapter 10

CHAPTER 10An Introduction to

Automated Software Testing

OUTLINE• What is Test Automation?• The Benefits and Pitfalls of Test Automation• When to Use Test Automation• Test Automation Production Process• The Evolution of Test Automation

CHAPTER 10

OBJECTIVES• Present an overview of test automation engineering• Understand the pros and cons of test automation• Test automation strategies, tactics and implementation

CHAPTER 10

An Introduction to Automated Testing

10.1 What is Test Automation?10.2 The Benefits and Pitfalls of Test Automation10.3 When to Use Test Automation10.4 Test Automation Production Process10.5 The Evolution of Test Automation

CHAPTER 10.1

What is Test Automation?

• Using computer time to execute of tests.• Commonly, test automation involves automating existing and

sometimes, new tests that otherwise, will be executed by a human tester

CHAPTER 10.1

• People might say, “Automation will help us release a higher quality product on time.”– What does that mean?

• What’s the goal of instituting an Automation program? – What problem are we trying to solve?

Why Automate Testing?

– Not enough test time in schedule for repetitive tasks– Very repetitive tasks– Too many releases to test– Better use of qualified in-house testing resources– Reduce cost of testers– Can’t test manually– Optimize tester time because there is not enough time for mundane

tests– Human tester can focus on testing that requires complex analysis– Save money– Save time


8

– If I automate testing, what manual testing will I lose? How many bugs might I lose and when?

Remember: Bugs are more expensive to fix later.


– Often, if the manual test team also does automation, it is the best or most technically talented testers who script. – The early part of application testing is when the most talented testers are

needed. • This is particularly true with script maintenance. Some talented

person on the main product team is often in charge of maintaining the automation project. – Will we lose finding good early bugs because of this?

– Testing cannot be an “on the side” job. You can’t write good scripts while you wait to get a build or have spare time!


Sometimes, the goal of a test automation program can be Good-Enough Automation. Just like with the products we sell, quality is a combination of reliability, features and timeliness. If your goal of automation is tightening the testing cycle. AND you can also improve the test battery, you have done a good job.

Goal of Automation

Manual vs. Automation Testing

Manual Testing Automation Testing

Test execution time Regression test coverage

Test execution time Regression test coverage

Responsive and flexible Relatively inflexible

Human variation Machine consistency

No maintenance needed High maintenance needed

Immediate dividend paid Long-term dividend

Not cost effective for high volume regression

Not cost effective for low volume regression

CHAPTER 10.1



CHAPTER 10.2

Test Automation Benefits

• Run automated tests unattended (overnight)• Reliable, consistent repetitive testing• Increased speed in test execution• Improves quality, increased test coverage• Reduces costs and time for regression testing• Optimize the use of test resources to focus on bug findings• Reusable and scalable if architected effectively• Execution of tests that can’t be done manually• Performance testing to assess software stability, concurrency,

scalability….

CHAPTER 10.2

Test Automation Pitfalls

• Uncertainty and lack of control.• Poor scalability and maintainability.

– Too sensitive to changes– Poor reusability

• Low test automation coverage– Too much programming

• Poor methods and disappointing quality of tests– Focus of tools instead production volume– Focus of making the tests run instead of quality of the tests

• Technology vs. people issues– Gaps in platform technology support– Gaps in human resource equipped with test automation skills

CHAPTER 10.2



CHAPTER 10.3

Which Test Cases Can’t be Automated?

• Usability testing"How easy is the application to use?“

• One-time testing• “ASAP” testing

“We need to test NOW!”• Ad hoc/random testing

Based on intuition and knowledge of application• Tests without predictable results

CHAPTER 10.3

The Criteria to Apply Test Automation

• Customer requests• Application should be stable• Tests that need to be run for every build of the application (sanity

check, regression test)• Tests that use multiple data values for the same actions (data driven

tests)• Tests that require detailed information from application internals

(memory, CPU)• Tests that require considerable amount of time to perform manually• Stress, load or performance testing

CHAPTER 10.3



CHAPTER 10.3

Manual

Input user actions

1

Wait for processing to complete

2

Check AUT function outputs

3

Repeat steps until all to-be-executed tests are completed

4

Automated

Generate automated scripts

1

Synchronize script playback to application output from each step 2

Add checks

3

Run tests or suites of tests

4

Test Automation Production ProcessCHAPTER 10.4



CHAPTER 10.5

A Brief History of Test Automation

Software Test Automation has evolved through several generations, with many improvements along the way:

1st Generation: Record and Playback2nd Generation: Scripting3rd Generation: Data-driven4th Generation: Keyword-driven

CHAPTER 10.5

Record & Playback

The automation tool records the steps of the user executing a test, and the tester configures some checks.

The automation tool generates a script based on the recorded actions.

The automation tool performs a playback of the recorded test. Even small changes to the application under test require manual

re-recording of the test. → This first generation is neither efficient nor scalable.

CHAPTER 10.6

Record/Playback

Log

Recorded test scripts

Test Tool(Invisible to the target system)

Test data entered by testers or users

• One time recording of actions and checks

• Multiple time playback

• Actions• Checks• Results

• Actions• Checks

TargetSystem/AUT

Playback

PlaybackRecord

EXAMPLEA Recorded Script

select window "Logon"enter text "username", "administrator"enter text "password", "testonly"push button "Ok"select window "Main"push button "New Customer"expect window "Customer Information"select field "Name"type "Jones"select field "First Name"type "John"select field "Address"type "54321 Space Drive" . . .

EXAMPLEA Recorded Script

• Dependent on screen coordinates

output14

# Analog Recording move_locator_track (1); mtype ("<T55> <kLeft>-<kLeft>+"); type ("<t3>output14" ); move_locator_track (2); mtype ("<T35><kLeft>-<kLeft>+");

mouse click

mouse movement

pause beforemouse click

enter textfromkeyboard

CHAPTER 10.6

R/P Pros and Cons• Pros

– A useful way to learn about the testing tool and automation

• Cons– High maintenance cost dealing with changes in the target

system of AUT– Poor accessibility to test cases – Greatly affected by the working system or environment– Only suitable for GUI systems

Scripting Approach• Automation is regarded as a programming activity• Parameterize hard-coded values• Separate data from code by moving variables to include files• Create utility functions to be shared• Produce and maintain like any other software• Train black-box test engineers to run the scripts

Scripting Approach• Pros

– Reusability of scripts for common tasks– Improved maintainability

• Cons– Poor accessibility to test cases because tests are

programmed or scripted– Greatly affected by a working system or environment– Test implementation must be done by programmers– Translation of test cases—3000 tests mean 3000

automated scripts– Manageability is still an issue due lack of visibility

EXAMPLEFunction EnterCustomerJones

LogonPress "New Customer"Enter Field "Name", "Jones"Enter Field "First Name", "John"Enter Field "Address", "54321 Space Drive". . .Logoff

End Function

Function OrderProductLogonPress "New Order"Enter Field "Product", “OurProduct 1.0"Enter Field "Amount", "35"Enter Field "Delivery", "asap". . .LogOff

End Function

. . .

Scripting – Example Script

Silk Test Script:

CHAPTER 10.6

(Scripting) Data-Driven Approach

• Take advantage of tester’s familiarity with test case design and creation using tables and matrices

• Accommodate localization projects• Recognize the importance of patterns in test cases• Enable testers to catalog test cases with spreadsheets in

software such as Excel• Enable testers to specify expected results in spreadsheets

GOODMORNING

goodmorning

GoodMorning

Input Data

EXAMPLE

• Text file used in the test:

• Data driven script:

• File with test data:nr case whole pattern matches1 off off GOODMORNING 42 off on GOODMORNING 33 on off GOODMORNING 14 on on GOODMORNING 15 off off goodmorning 46 off on goodmorning 37 on off goodmorning 18 on on goodmorning 19 off off GoodMorning 4

10 off on GoodMorning 311 on off GoodMorning 212 on on GoodMorning 1

goodmorningGOODMORNINGGoodMorningGoodMorningVietnam

for each line in the file do open the find dialog read a line from the file use the values to fill dialog press find button check amount of matches close the dialog

EXAMPLE

• Pros– Separation of action and data enables tests to receive

combinations of test data– Test data can be stored in a separate file or spreadsheet for better

maintainability– Improved reusability and maintainability

• Cons– Actions containing business logic are in automated scripts– Maintainability of changes in business logics as well as interface is

still poor– Poor visibility because accessibility to the tests is still poor– Not scalable (e.g., reaching 95% of automation)

(Scripting) Data-Driven Approach

Test Automation Challenges

• Too much programming• Response to rate of changes (software)• Low reusability/scalability• Too many platforms/technologies• Too expensive if the returned value is low

Keyword-driven

Tests are broken down into a series of actions representing the business processes of the SUT, such as ‘enter client’, ‘transfer’, and ‘check balance’.

Individual actions are automated, not the tests. The tests are stored externally from the test scripts, typically in a

spreadsheet. Keyword-based test automation’s elegant, structured approach

simplifies the work even further, reducing the time and expense of test design, automation, and execution by allowing team members to focus on their areas of individual expertise.

Non-technical testers and business analysts can develop executable test automation from keywords representing recognizable end-user actions.

CHAPTER 10.6

How Does ABT (Action-keyword) Work?

Test Cases are written or described through a series of Test Lines using parameterized and reusable Action keywords

Actions are created and communicate with target systems and software applications including the application under test (AUT) through one or more supported Interfaces to carry out intended tasks

Note: For now, think of an Action as a verb or something that you do

Information about Interfaces such as Windows, Web or Java graphical user interface (GUI) or command-line interface (CLM) are later defined to enable actions to send inputs to and receive outputs from the target system and/or AUT

Test Line

Action keyword

Action’s Parameters

Number Name Descriptionadd product 12345678 Hammer For pounding nails

• Separation of action, interface, logic and data• Test data can go in a separate file or spreadsheet• Test logic can go in a separate file or spreadsheet• Interface can go in a separate file or spreadsheet • Action can also go in a separate file or spreadsheet• Action can be reused

Overcome Challenges w/ Action Based Testing

Automation Approach Comparison

Approach

Workload Allocation

Automation Engineering

Test Case Production

Test Data Construction /

Selection

Record & Playback N/A Test Engineer N/A

Scripting Automation Engineer

Automation Engineer N/A

Data-driven Automation Engineer

Automation Engineer Test Engineer

Keyword-driven Automation Engineer Test Engineer Test Engineer

CHAPTER 10.6

Note: Programming skill is required for Automation Engineer

Examples of Test Automation Tools

AutomationTool

HP IBM Others

Functional WinRunnerQuickTestPro

Functional TesterRational Robot

SilkTestTestPartnerTestComplete

Performance LoadRunner AstraLoad Test

Performance Tester SilkPerformerWebLoadOpenSTA

Management Test DirectorQuality Center

Rational TestManager SilkCentral Test Manager

Memory Purify JProbe Profiler

LogiGear is the producer of TestArchitect™, a functional automated testing tool designed to help achieve the most efficiency out of action-keyword test automation. Examples of other commercial tool vendors include:

CHAPTER 10.6

SUMMARY• Test Automation is a method that uses of software to control

the execution of tests.• Commonly, test automation involves automating a manual

process currently in.• Test Automation has benefits such as…However, it still has

some pitfalls such as…• A effective strategy of test automation includes several steps…• There are many popular automation tools. Depend on the

AUT, we choose the suitable automation testing tool.

CHAPTER 10

QUIZ• What are the advantages and disadvantages of test

automation?• What are the advantages and disadvantages of test

automation generations?

CHAPTER 10

Education

Chapter 10