Ensuring Application Reliability and Scalability Through Effective Test Automation

®

IBM Software Group

© 2004 IBM Corporation

Ensuring Application Reliability and Scalability Through Effective Test Automation

By Denice Wong

Technical Consultant – IBM Rational Hong Kong

IBM Software Group | Rational software

2

Challenges Associated With Testing

Insufficient/Changing requirements

Cau

ses

Inaccuratetests

Waterfallprocess

PoorUnit testing

Limited timefor testing

Poor Performance testing

Late detectionof errors

Functional testingblocked

Unable to perform full regression testing

Inaccurate performance prediction on

production environment

Effe

cts


3

a

An Example: Challenges of Siebel Implementations

Cost of testing new Siebel installation

$0

$100,000

$200,000

$300,000

$400,000

$500,000

New Installation

Simple Complexity Installation

Medium Complexity Installation

High Complexity Installation

Cost of testing new Siebel installation

$0

$100,000

$200,000

$300,000

$400,000

$500,000

New Installation

Simple Complexity Installation

Medium Complexity Installation

High Complexity Installation

Surveyed 130+ projects

Results: Typical customer spends ~25% of

project costs on testing

Testing costs vary widely depending on project type and complexity

“Customers are spending close to $½ Million on testing complex projects”1

1 Source: Siebel Test Automation Datasheet


4

Do you ever have enough time to test everything???

Trade coverage for risk Reduce scope of testing

Reduce testing cycles

Reduce configuration testing

Add people to the project

Introduce / enhance automation


5

Automation Adds Options!

Improve human productivity

Broader test coverage

Reduce redundancy

Automate repetitive tasks

Enable process improvements

Increase regression testing

Expose risk areas

Tie testing to requirements


6

What Does Test Automation Mean?

Do my testing for

me!


7

What is Automated FunctionalTesting?

Using automation tools to record actions taken against an application.

Test scripts are derived from these recordings which are then used for subsequent test execution.

Record

Enhance

Execute

Record

Enhance

Execute


8

What Is Automated Performance Testing?

The process of exercising an application by emulating actual users with a load generation tool for the purpose of finding system bottlenecks

Controller

System Under Test


9

What is Automated Test Management?

Using automation tools to accelerate the process of test planning, test creation, test execution and test results analysis

PlanPlan

Create

ExecuteExecute

AnalyzeAnalyze


10


11

Challenges of Functional TestingChallenges of Functional Testing

Controlling cost of test automation Inexperienced testers face steep

learning curve for tools

Test maintenance outpaces testing

Controlling cost of test automation Inexperienced testers face steep


Test maintenance outpaces testing

“Our testers are not experienced in building

automated tests.”

“Application UI changes are always

breaking our test scripts.”


12

Challenges of Functional TestingChallenges of Functional Testing

Control cost of test automation Inexperienced testers face steep


Test maintenance grows with project

Testing Tools Lack Flexibility Power users hands are tied by tool

limitations

Control cost of test automation Inexperienced testers face steep


Test maintenance grows with project


limitations

“I need one tool for both my expert and

novice testers.”

“Our script language limits our ability to overcome some

automation obstacles.”


13

IBM Rational Functional Tester Automated regression testing for Web,

J2EE and .Net applications

Key Benefits Minimize test maintenance with scripts

resilient to application changes

Wizard enhanced automation to speed test creation for the new user

Powerful scripting language and IDE for the professional tester

Supports Team oriented parallel development

Functional Testing with IBM Rational Functional Tester Test automation for the novice and the professional


14

Execute

Effective Test AutomationCreating an automated test is a three steps process

Script Recording Capture user interactions with the system under test

Script Enhancing Modify scripts for situation specific challenges

Script Execution Ensure reliable playback for local or remote execution

EnhanceRecord


15

Data Driven Testing Separates test data from test

script

Enables a single script to run multiple tests by using multiple data sets

Wizard driven process No programming involved

Import data from external sources

Data Driven Testing Separates test data from test

script

Enables a single script to run multiple tests by using multiple data sets

Wizard driven process No programming involved

Import data from external sources

Recording ScriptsData Driven TestingRecording ScriptsData Driven Testing


16

Dynamic Data/Content Matching Use pattern matching technique to verify dynamic data and create robust tests

E.g. Instead of validating “Order ID 230”, validate “Order ID ###” or Order ID 2##, etc.

This allows for a wide variety of acceptable responses as well as restrictions on acceptable responses when validating the application’s behavior

Order ID changes with each order placed

Order ID changes with each order placed

Test Fails when

comparing static

baseline to dynamic data

Test Fails when

comparing static


Test Passes when

comparing pattern


Test Passes when

comparing pattern


Recording ScriptsValidating Dynamic Data


17

Functional Tester offers two development environments Eclipse based IBM Software

Development Platform

Visual Studio .net

Both environments offering powerful debugging features Code assist editors

Step debugging

Variable watches

More…

Functional Tester offers two development environments Eclipse based IBM Software

Development Platform

Visual Studio .net

Both environments offering powerful debugging features Code assist editors

Step debugging

Variable watches

More…

Enhancing ScriptsPowerful, Professional Debugger


Reduce Test Script MaintenanceObject Map

The object map is a collection of descriptions of test objects

It is automatically populated during test script recording

It contains properties for each test object Recognition properties enable Rational XDE Tester to find the test object

during playback

Changes to an object need only be made once – in the map

Central Object Map

EditBox

CheckBox

Button

Frame

Test ScriptsTest ScriptsPanel


19

Reduce Test Script MaintenanceReliable Playback with Script Assure

DeterminesMatch

Version 1.0 Version 2.0

TesterSees

ToolSees

No User No User Intervention Intervention

Required With Required With ScriptAssureScriptAssureTMTMScore = 75


20

Broad Environment Support Web, Java. .net Create data driven tests without coding Static data and properties verification Dynamic data validation without coding What can’t or shouldn’t be automated?

Flexible and Extensible Scripts Flexible coding language Powerful, professional debugger Object map editing flexibility Version control of scripts

Script Execution Options Reliable playback with ScriptAssure Remote and local playback on various platforms

Broad Environment Support Web, Java. .net Create data driven tests without coding Static data and properties verification Dynamic data validation without coding What can’t or shouldn’t be automated?

Flexible and Extensible Scripts Flexible coding language Powerful, professional debugger Object map editing flexibility Version control of scripts

Script Execution Options Reliable playback with ScriptAssure Remote and local playback on various platforms

Summarizing Key Features in IBM Functional TesterSummarizing Key Features in IBM Functional Tester


21


22

Challenges of Performance TestingChallenges of Performance Testing

Complex Tools & Complex Task Tool complexity and lack of

experience intimidates many first time users



“We don’t have a lot of experience building

good load tests.”

“Most load test tools are much more

complicated than we require.”


23





limitations




limitations

“We don’t have a good understanding of what

exactly is occurring during a load test.”

“We have highly technical performance

testers that need to customize their tests.”


24





limitations

Hidden Costs Hardware intensive tools and inability

to share information with team mates increases cost of testing




limitations

Hidden Costs Hardware intensive tools and inability

to share information with team mates increases cost of testing

“Our developers have no way of performing

load tests.”

“We don’t have enough hardware to run our

tests.”


25

IBM Rational Performance Tester Performance Testing for Web Based

Applications

Key Benefits Built for Day 1 Productivity

Mask complexity to get the job done

Advanced Data Access & Manipulation Automated data variation and

synchronization

Simulate real world workload model

Lower Cost of Performance Testing Low resource requirements

Desktop IDE integration

Performance Testing with IBM Rational Performance Tester Test automation for the novice and the professional


26

Execute & AnalyzeSchedule WorkloadBuild Scripts

Script Creation Considerations Visual test editor, varying input data & correlating server responses

Scheduling Considerations Accurately representing a true user workload

Execute and Analyze Considerations Validating responses & finding the bottleneck

Creating a Performance TestCreating a performance test is a three step process


27

No-Code Tests

Tests are represented in a tree view as a list of web pages and their elements, such as images.

No programming necessary to create a test

Underlying detail is accessible but hidden to simplify test creation and maintenance


28

Creating a Performance TestBuilding Scripts

Automated creation of parameterized tests Literal values used during recording are replaced with variables

At playback time, variables are populated with data from a datapool

Data for playback can be imported from external data sources

No user coding involved

User Name: JSmith

User Name: CBryson

User Name: TJones

Session No. 100

Session No. 101

Session No. 102


29

LoadSmart schedulingEasy to use

Model complexreal world scenarios

Point and clickscheduling


30

Creating a Performance TestSchedule Workload

Powerful and flexible scheduling Visual schedule editor for no code scheduling

Coordinate timing and dependencies

Accurately model real users workloads

Dynamically increase the load during the run


32

Find Slow Pages

Easily determine slow performing pages of your web site Reports are available live during the run and for post run analysis


33

Find Resource Bottlenecks

Simultaneously monitor resource utilization during the test Capture and analyze a long list of performance monitoring statistics

Determine your resource bottlenecks


34

Identify the Root Cause of Performance Problems IBM Performance Optimization Toolkit for Rational Performance Tester

System System underunder test test

Run Performance Test to Run Performance Test to Emulate Multiple UsersEmulate Multiple Users

AnalyzeAnalyzeApplication Application

Response TimeResponse Time

Isolate elusive Isolate elusive performance performance problems in problems in complex, multi-tier complex, multi-tier environmentsenvironments

Speed delivery Speed delivery of quality softwareof quality software

aa

bbIsolate Isolate

Performance Performance Bottlenecks and Bottlenecks and Submit Change Submit Change

RequestRequest

DeveloperDeveloperDeveloperDeveloper

TesterTesterTesterTester

cc

Pinpoint performance Pinpoint performance problems in the test lab...problems in the test lab...

Speed changes by Speed changes by providing detailed providing detailed data to developersdata to developers

UserUserLoadLoad

11 22


36

Summarizing Key Features in Performance Tester

Built for Day 1 Productivity Create code free tests

Mask complexity to get the job done

Advanced Data Access & Manipulation Create code free schedules

Automated data variation and synchronization

Create real world workload model

Lower Cost of Performance Testing Low resource requirements

Desktop IDE integration


37


38

Why Should I Test?

Associating Test Inputs defines Why you are testing Establishes a link between the test and the motivator

Motivator can be RequisitePro Requirements, Rose Model, Excel Spreadsheet, or custom

Establishing link enables requirements coverage reporting

Test InputTest Input


39

Suspect Test Cases

Suspect Test Cases show up with red slashes in the Test Inputs View

Suspect Test Cases show up in bold in the Test Plan View

AND

Within TestManager:


Automatically Track Impact of Changing Requirements

Automatic Notification of Changed

Requirements that Affect Test Cases


Comprehensive Change/Defects Management

Test Case

Script

Requirement


Control Your ProcessTrack Test Thoroughness

Requirements (ReqPro)

Test Case (Test Manager)

Defect Tracking (ClearQuest)


Control Your ProcessTrack Test Thoroughness


44

Summarizing Key Features in TestManager

Speed your Test Cycles TestManager controls, tracks and

maintains all test assets

Simulate Real World Workload Create real world workload model to

make sure the test result is meaningful

Effortless Analysis Intuitive reporting provides results

analysis for accurate and up-to-date test reports


45

CustomerExtensions

3rd PartyISV Tools

Rational Software Architect

Rational Application Developer

Rational Functional & Manual

Tester

Rational Performance

Tester

RequisitePro, ClearQuest, ClearCase, Test Manager

Tivoli Configuration

Manager

RequisitePro

Rational Software Modeler

TivoliMonitoring

Rational Web

Developer

Project Manager

ExecutiveRational Portfolio Manager

The IBM Rational Software Development PlatformA complete, open, modular, and proven solution for Software Quality Assurance

AnalystAnalyst ArchitectArchitect DeveloperDeveloper TesterTester Deployment Manager

Deployment Manager


46

What’s Next?

Rapid Web Application Deployment Seminar (26th May, 2006)

Please fill in the feedback form and get IBM Rational evaluation CD kit !!

Documents

Ensuring Application Reliability and Scalability Through Effective Test Automation