Optimizing the Test Effort With Keyword-Driven Frameworks

8/14/2019 Optimizing the Test Effort With Keyword-Driven Frameworks

1/46

Optimizing the TestingEffort with Keyword -

Driven FrameworksBrian Massey, IBM

March 31 April 2, 2009


2/46

Enter Text/ g raphic s here

Enter Text/ g raphic s hereEnter Text/ g raphic s here


3/46

Quic k Bio

Product Manager No worries not here to sale you anything (but to ensure the kids

get fed I have to mention them)

As part of Rational Services organization helped develop and deployframeworks for our customers

Established Software Quality Centers of Excellence for Companies

Experience as Developer, Tester, Architect, Dev Manager, QA Manager

Used to build Air Defense Systems for a living Developed and tested embedded systems

Father of 6 so prepared for anything


4/46

Objectives

This class presents a framework for keyword-driven testing.

How to easily plan and create manual tests using aframework

Youll learn how keyword bridge manual testing toAutomated testing


5/46

Agenda

Identifying the p rob lem

The Basics Testing Frameworks how and why?

Solutions in the ind ustry

How will Keywords help me Pro and Cons of Keyword testing

Assessing Automation Suc c ess Metrics


6/46

The Prob lem Companies want to ultimately achieve Test Automation

Faster release cycles achieved due to concepts such asAgileUI not stable for automation early onHave to compress test cycle

80% of Testing still done Manually

Most of the skills in any given Test Organization are ManualTesting

Companies want to see fast return on Investment for Test

Automation solutions


7/46

Skills Separa tion

60%

30%

10%

App lic a tion Experts Plan a nd Design Tests

Ma nage Test Data

Implementers Built sc rip ts based on modules and

utilities

Coders Write modules and utilities

Could be develop ers on loa n


8/46

Coc kp it Alignment Prob lem

Proper Alignment with Each Testers SkillsetTest Automation

traditional record/playbackcaught in the middle

Manual Testing


9/46

Why is Keyword Driven Testing

Needed? More productivity desired from the non-technical base of users

Test automation tools need to be easier to use for business analysts andsubject matter expects

Details of scripts are hidden from the user Users dont have to learn the script language to write scripts

ROI

Companies want faster ROI on purchasing of Automated Test Tools Companies want investment is test professionals to provide quicker

returns

Need to reduce the cost of developing and maintaining scripts

Need to improve script resiliency

Reusability

Common script development techniques


10/46

The Prob lem Where is the Easy Button

IBM


11/46

Agenda



Solutions in the ind ustry




12/46

The Basic s What is a

Framework

Keyword works with an automation framework but just

using a framework does not make Keyword Driven testing

Frameworks Frameworks are simply Function Libraries extending the

Automated test tool being used. Libraries, functions, etc created to interface with Applications. Automation Scripts utilized these Libraries Very proprietary to the organization and often to the AUT Do not resolve the usability from the non-technical tester First step to the evolution of Keyword Driven testing


13/46

The Basic s What is aKeyword A Word that provides a simplified

representation of a more complexconcept

E.g. Keywords Encapsulates Business Logic Windows Objects

Screen Functions Services

Thus, a Keyword is A logical

representation to one of thecomponents above

Keywords need to be properlydefined for Automation Experts to

develop proper framework


14/46

Many Names in the Market

Many Names one Concept

Action words / Actions Based testing (Hans Buwalda, LogiGear)

Test Frameworks (Linda Hayes, Worksoft)

Third-generation test automation (Edward Kit)

TestVerb technology (TestQuest)

Keyword Driven Testing/Business Process Tester (IBMRational/Mercury)

Table Driven Testing

Data Driven Testing


15/4615

OK show me a p ic ture!!

AUT

AutomatedTest Tool

ApplicationObject map

TestAutomation

Application

Framework

Keyword

Test

Keyword TestParser

Libraries todrive keywords

Keyword TestScript

Keyword Map

UserCreatesUses


16/46

The Basic s Keyword Driven

testing

Keyword Driven Testing provides a mechanism for mapping

an Application Under Test (AUT) to a set of defined and wellunderstood keywords. Allows testing of an application by interfacing only with the

keywords.

Underlying Automated tool at a different level of abstraction AUT is at a different level of Abstraction

Division of labor between Application Experts (BAs, SMEs, etc) and

test automation experts

Creation of modular, reusable test components that are built by testarchitects and then assembled into test scripts by test designers


17/46

Why Keyword Driven Testing?

Provides Clarity

Teamwork andoptimization of

resources Working Smarter not Harder


18/46

Agenda



GUI Automation 101

Ma king Automation too l Bet ter

IBM Examp le Solutions in the ind ustry




19/46

What To Automate

Tests tha t need repea t exec ution

Tests tha t c over existing , stab lefunctionality

Tests tha t rep resent simp le designs

Tests tha t offer numerous opportunities

for da ta input varia tion


20/46

GUI Automation 101 Importa nt Things to Know Test Automation is a SOFTWARE PROJECT

Arc hitec t, p lan and develop your c od e for the long term

Writing good autom ation is c ha lleng ing but worth it

GUI automation is a spec ia l skill set and a c areer nic he!

Automation is a long-term investment w ith extended ROI

It WILL take longer to c rea te an autom ated test c ase than a ma nual test c ase

It typ ic a lly ta kes 3 release c yc les to see substa ntia l ga ins from automating atest c ase

Automation need s buy-in and help from develop ment

Automation b loc king defec ts need to b e fixed , even if customers won t see

them

Knowing a bout app lic a tion c hanges that b rea k automa tion in advanc ereduc es fire d rills for the test exec ution c ommunity

Having developers run automation prior to c hec k-in rea lly improves c ode

stability

Automation ensures todays build works like yesterdays and c ana llow la te c hanges to the p rod uc t


21/46

GUI Automation 101

Importa nt Thing s to Know Applic a tions Und er Test WILL c ha nge.

Your automation needs to be a rc hitec ted for ma intenance

Separa te whatyour app lic a tion does from how it d oes it

Keep the GUI definitions in a sing le p lac e

Understa nd your app lic a tion, besides GUI c ha nges, wha t else ??

Know tha t GVT testing is c oming externa lize strings for lookups

and verific a tions or build the support in right away

Rec ord / p laybac k is impossib le to ma inta in release to release

Isola te GUI ta sks as func tions or c lasses tha t work on a sma ll

portion of your app lic a tion Dynamic Verific a tion reduc es ma intenanc e: Use your test d a ta

to figure out wha t to verify.

Naming c onventions and pac kage struc ture c an rea lly help as

your p rojec t gets more c omplex

Define your development p rac tic es Rules of the Road (ROTR)


22/46

IBM I t l F k B fit


23/46

23

IBM Interna l Framework Benefits

Out-of-the-box business-log ic based framework on top of RFT

Red uces tota l c ost o f ownership for mixed tea ms tha t want to d eve lop aframework tha t enc apsula tes the ir business log ic

Jumpstart Agile a nd/ or la rge a utomation projec ts Simplifies coding tasks with easy-to-read lib ra ry c ode and standard ized

implementation

Re-useable code for multi-inte rfac e app lic a tions (Web, Ec lipse & API)

Business Logic reduc es design w ork; integra tes with existing RFT Keywords solution

End -user va lida ted

IBM interna l users a re c ontinua lly adop ting , enhanc ing and build ing new fea turesfor the frameworkc od e b ase


24/46

24

Framework Methodology

Build automation foc used on the app lic a tion s business log ic

Ca n be used with keywords

Fac tory imp lementa tion c an support multip le c lient interfac es withmaximum c ode re-use (graphic on next slide)

Even sup ports testing API, CLI

Provides easy to read test c ases

Allows non-programmers to write test c ases

Keywords/ methods a re mea ning ful to Doma in Experts


25/46

25

Framework Arc hitec ture

Re-useable codefor all clients

Re-useable codefor all clients


26/46

WAIT -- Keyword Automation

When Not To Use Keyword

Automation Fringe Ac tivities where reuse potentia l is

low

Complex Tasks where other frameworksand/ or approa c hes fit better(e.g. Testing API Interfac es, etc )

Lac k of d ed ic a ted automation resourc es

K d/ F k S l ti


27/46

Keyword / Framework Solutions

Keyword Context (in Keyword Driven Testing) has been created for around 10 years,working as:

Standardized Interface in Native language Across Different Contexts (such as, manual testers,automation developers),

Mapping to a unique, reusable, Function in programmingAble to Share Across Teams/Projects, SUTs

QTP/BPTWhy Certify RQM/RFT HomegrownSAFS

The Real Point for Keyword Driven Testing is at Why, not at What-is

Why Keyword?

Aim to Deliver an Effective Test Automation Solution

Seamlessly Integrate Expertise on All Ends of the whole lifecycle, including From Business Analyst (Expertise on User Scenarios) From Manual Tester (Expertise on Product & Environment) From Automation Developer (Expertise on Scripting/Programming)

High-levelComponent

Keyword


28/46



First, Create .Map file. This is a set of keywords and how they mapto your Functional tester Objects


29/46



Step Table (.SDD Files) tells functional tester what to do for each step as itreads the application map file. It is the Test

Suite (.STD files) group Step files together


30/46



Advantages Enables non-technical

testers create tests using an

application such as notepadbased on keywords definedin the map file.

Tests can be created quickly

and easily No knowledge of underlying

automation framework isneeded

Disadvantages A lot of setup required for

mapping keywords

The framework requirescustomization for eachautomation testing tool

Not very fancy, have to

maintain things in text format. Lot of work needs to be done

external to the tool Keywords need to be mapped

to objects in the automationtool


31/46


Mercury EULAYou may not...provide externally or to thirdparties any oral or written communication

that describes or summarizes the features,functions or performance characteristics ofthe Licensed Program and Documentation,or that compares the Licensed Programwith any other similar product of Yours orany third party;



32/46



3 components to Worksoft Certify GUI Client to plan, design, execute and maintain tests Central DB server for maintaining all assets

Function Library (e.g. Framework)

Manual Tester/End User creates a test from an applicationmap

Application Map CreatedUsers specifies the object and operation from the map and any data

needed for a sequential flow through the application Specify actions that can happen on each step (e.g. On failure

continue executing, end test, Jump to another step, etc) If objects in application map changed test steps using changed

objects are flagged Test results log maintained in DB


33/46



3 components to Rational Quality Manager Solution Manual test tool used to creation and execution of manual tests leveraging

Keywords

Automation tool for creation and execution of scripts to tie to keywords Keywords that represent application Business Logic

Manual Tester/End User creates a Manual test Keywords defined in RMT

Framework (Automation scripts) built in RFT and tied to keywordBased on what is scoped by keyword definitionApplication logic captured here

Users build Manual tests leveraging keywords

Specify Data to use in the Manual test Script for variableized content


34/46


The Real Point for Keyword Driven Testing is at Why, not at What-is

QTP/BPTWhy MTP v3.5 RQM/RFT HomegrownSAFS

One Customer Example

Goal Make Scripts simpler to read and maintain Want scripts to only look like Manual tests Improve ROI

Develop a manual test Create external object maps maintained in CSV file.

Consists of Object, Action, Value. (Use test Object Inspector to find object names) Assumption is that object names dont change

Hidden properties of Manual Test Step to designate the object, action, value on each step Execute Automated Test where all testcases execute same generic automation script Have a Automation Framework

Parses each line of manual test and extracts the object and operation desired

Calls library files passing the object and operation on the object to drive application. Manual test tool used to creation and execution of manual tests leveraging Keywords Automation tool for creation and execution of scripts to tie to keywords Keywords that represent application Business Logic


35/46

Commona lities ac ross solutions

All solutions require c rea ting a map of the AUT

Some work with an app lic a tion ma p d irec tly

Others map Keywords to the Test Tool ob jec ts and / or Transac tions

Test Tool Ob jec ts still mapped to AUT ob jec ts

Cod ing is required on the b ac k end , espec ia lly for c ustom c ontrols

Ap p lic a tion Experts need to know the Keywords and types ofinterac tions that c an oc c ur on the Keywords

Even if a keyword is mapped to the ob jec t, c ustomers need toknow the keyword and op era tion on the keyword

Faster ROI is a p rimary d rive r

Testers want to justify the ir purc ha ses and existenc e

Primary outreac h is for the Business Ana lysts or Non-tec hnic a l Teste r

Want to utilize the w ork being done by the BA s and make toolseasy enough for a ll typ es of users

N d A b l d h


36/46

Need A ba lanc ed approac hMeet the needs of the b usiness in a c ollabora te

manner

Streamline Manua l Testing

Crea te living self-mainta ining d oc umenta tion

Red uce d ep endenc y on subjec t ma tter experts Minimize inc onsistenc ies of manua l testing

Business Relevant Automated Testing

Flexib le and sc a lab le framew ork

Demonstra te va lue quic kly

Agility To Meet Business Needs

Strateg ica lly utilize sta ff and the business

Maximize investment in manua l test assets Minimize a utomation maintena nce over time

Keep the entire test team on the same page


37/46

Agenda


The Basics

Testing Frameworks how and why? Solutions in the ind ustry




38/46

3838

Testing done earlier in lifecyc le Define Tests before app lic a tion sta b ilized Inc reased ROI on Testing Tools Create Tests from spec ific a tions earlier in the Software LifeCyc le

Bridge to Test Automation Use App lic a tion experts (Business Ana lysts, SMEs) for testing

Use c ommon terms to define test c ases and sc ena rios ea rlier in the p roc ess 80% testing done manua l op timize tha t investment

Benefits of Keyword Testing

Manual Tests Test AutomationSteps tiedto

keywords

Keywordsrea lized byautomation

Ke ord Testing Go ls


39/46

Keyword Testing Goa ls

Crea te manual testseasily

Automate p iec es of atest Have hybrid ma nua l-and-

automated sc rip ts

Assemble the p iec es intonew sc rip ts

Crea te fully automated sc riptswithout cod ing

Let an automation

expert automate thepieces


40/46

Keyword Automation w ithin Ag ile Proc esses

Release Eng ineering / Build Proc esses

Automation is c onsidered a best p rac tic e

Quic ker detec tion of regressions with eac h

build

Func tiona l Test

Automa tion results c an a llow deve lopment

teamsto reac t faster to defec ts and fulfill the ag ilevision

Pitfalls

Automation is very d iffic ult w ith 1.0 Projec ts

Higher level of code c hurn whic h inc reasespressure on up front investment


41/46

Metric s to c ollec t

Things to und ersta nd to bette r eva lua te our ROI inautomation:

How Long

How long it ta kes to run the tests manua lly

How long it takes to ana lyze the test results (ranma nually and ran automa ted)

How long it takes an unma nned ma chine to run theautomated suite or tests.

How Many

How many tests we p lan to automa te (an ac tual count)

How ma ny tests we ac tually have automa ted -- a t givenintervals

How many resourc es invo lved in running the testsmanua lly (inc lud ing setup , c lean-up a nd test ana lysis)

How ma ny resources involved in running the automatedtests (inc lud ing setup , c lean-up and test ana lysis)

Note:

Automation isn't a lways the best a nswer. These m etric s will help determine the best c ourse o f ac tion.


42/46

Key Solution Drivers

Ease of Use

Utilize Manua l testing for Business Ana lysts easy to use she ll

Little to no p rogramming

(Automated Sc rip ts rec orded , or crea ted by test autom ation eng ineers)

Represent tests to d ifferent ro les in forma t they are most c omforta b le with (BA ssee tests in Na tura l Language)

High level manua l tests and keyword view

Faster c rea tion o f a rob ust, rep ea tab le a nd low ma intena nc e test Keyword ma pp ing and automa ted sc rip t genera tion

Ab ility to c rea te tests without an a pp lic a tion c rea te from spec ific a tions ea rlierin the Software LifeCyc le

Keyword Ma pp ing to ob jec ts and head less c omponents

Unattended p laybac k

Automated test assoc ia ted onc e manual test has been keywo rd ena b led

Quic ker Return on Investment fo r c ustomers buying our solution

Crea tion of tests earlier and faster

Reusability


43/46

Keyword Automation Summary

Not a silver bullet for a ll testing

Requires ded ic a ted resourc es

Keyword Automation Frameworks integra tedwithin the tooling help red uce the upfrontburden

Certa in types of testing have an a ffinityfor c erta in types of automation

High levels of keyword re-use andinc rementa l test coverage expansionind ic a te inc reasing benefit from keywordframework.


44/46

References

Nagle, C. Test automation frameworks ; 1999

Hayes, L. "Esta b lishing an Automated Testing Framework"Tutoria l p resented a t STAR99EastConferenc e, Orlando,Florida, May 11, 1999.

Nagle, C. Automation Framework Support , Ma y 2005

Cem Kaner, http://www.kaner.com

Zambelic h, K. Tota lly Data -Driven Automated Testing1998

http://www.sqa-test.com/w_paper1.html
http://www.kaner.com/http://www.sqa-test.com/w_paper1.htmlhttp://www.sqa-test.com/w_paper1.htmlhttp://www.kaner.com/


45/46


46/46

Thank you!

For more information:

Brian MasseySenior Produc t Manager

IBM/ Rationa l Software678-248-4523

[email protected]/rational

Documents

Optimizing the Test Effort With Keyword-Driven Frameworks