ANZTB 2010 – ConferenceHybrid Keyword Data Driven Automation Frameworks by Jonathon Wright

Tuesday, 2nd March 2010

© ANZTB 2010

Hybrid Keyword Data Driven Hybrid Keyword Data Driven

Automation FrameworksAutomation Frameworks

Test Automation as a Service (Test Automation as a Service (TaaaSTaaaS))

Real world case studyReal world case study

Jonathon Lee WrightJonathon Lee Wright

jlwright@iee.orgjlwright@iee.org

www.automation.org.ukwww.automation.org.uk

www.linkedin.comwww.linkedin.com/in/automation /in/automation

11th November 201011th November 2010

• Complete “Automation Lifecycle Management ” (ALM) solution;

• Providing “Automated Testing as a Service ” in the Cloud ;

• “Pay As You Go ” flexible billing (no expensive tool licensing costs);

• Distributed continuous build integration with “Application Under Test ”;

• Shorter market deployment times through quicker Automated Testing ;

• Shorter environment deployment times through Automated Health Check ;

• Better quality testing freeing specialist test resource to concentrate on non-repetitive test tasks;

• Easy interface for non-technical testers to create their own “user stories ”;

• Capability Maturity Model Integration (CMMI) through Business Process Modelling (BPM) comprising of Business Level Keywords (BLK);

• Introduction of “Full ” Agile automation testing and “First ” day automation.

What value did this case study deliver?

What approach?

• Manual Approach;

• Traditional Automated Approach:

• Hybrid Keyword Data Driven Framework Approach.

Vendor Tool(Record & Playback,

Adapt Scripts)Limited requirements Limited output

Framework Approach

Hybridutilising the best technologies and resourcesHybrid

Keyword

Data

Driven (Modularity)

Keywordbusiness level keywords represent user stories

Datadynamic use of sanitised business data

Driven (Modularity)modularity of test procedures to provide reliable processing of the framework components

Hybrid Approach

.NET 4.0 Development (Visual Studio 2010)

Cloud (Windows Azure/SQL/AppFabric)Automation (Ranorex/Coded UI APIs))

●●●●●Hybrid

Vendor Tool (Ranorex IDE)●○○○○Traditional

ToolsModularityDataKeywordBPTBPMApproach

○ - Limited Support

● - Full Support

Single Technology Stack (SPOF)

Tool Vendor Dependency

C# Project incompatible with

Visual Studio

Technology Blocks(Custom Flex Objects/Classes)

Limited non-technical tester

interface

Vendor Support(maintenance subscription)

Object Repository

(object-based Record / Replay)

ConstraintsAdvantages

Traditional (Vendor Tool)

In-house framework development & maintenance effort.

Additional skills / knowledge

resources

Azure Platform Subscription

Object Repository (limited Record / Replay support)

Multiple Technology Stack(VS2010 / C# / VB.NET / ASP.NET / TFS

Windows Azure / (WCF) Web Services / DLL/ XML/ Azure SQL/AppFabric

Coded UI / Ranorex / AIR / Flex / XPATH )

Integration with development (Visual Studio 2010)

Integration with deployment(TeamCity v5.1)

Integration with testing(Quality Center v11)

ConstraintsAdvantages

Hybrid (.NET Development)

Return on Investment

1 2 3 4 5 6 7

Number of Excecutions

Inve

stm

ent i

n E

ffort

MarketConfigurationTesting + HealthCheck

MarketConfigurationTesting

Traditional

Manual

MCT = 3 MCT+ HC = 3.4 Traditional = 3.7

Initial Investment

LeverageHybrid (Framework)

Vendor Tool

Rn = Aa / Am = (Va / n + n*Da) / (Vm / n + n*Dm)

ANZTB 2010 – ConferenceHybrid Keyword Data Driven Automation Frameworks by Jonathon Wright

Tuesday, 2nd March 2010

© ANZTB 2010

Automation Goals

Reusablemodularity of

test procedures & functions

Manageableeffective test design,

execution and traceability

Maintainableto reduce the amount

of test maintenance effort

Robustproviding

object/event/errorhanding & recovery

Flexibleindependent of

test environment and be scalable

Measurablecustomisable reporting

& test execution progress

Business Process Testing (Process)

Business Process Modelling (Example)

Index- Complete- In Complete- Supported - Out of Scope

Business Level Keywords (Testing)

Translates

Automation Lifecycle Management (Process) Technology Stack (TaaaS)

ANZTB 2010 – ConferenceHybrid Keyword Data Driven Automation Frameworks by Jonathon Wright

Tuesday, 2nd March 2010

© ANZTB 2010

Overview (TaaaS) Benefits (TaaaS)

Architecture (TaaaS) High Level (TaaaS)

Automation Lifecycle Management (ALM) Requirements - Business Process Data

AUT Database

Quality Center

Test Automation Database

Mock Live Data (Sanitised)

Team Foundation Server

ANZTB 2010 – ConferenceHybrid Keyword Data Driven Automation Frameworks by Jonathon Wright

Tuesday, 2nd March 2010

© ANZTB 2010

Test Planning - MasterDriver (Excel)

Object Repository

Test Repository

Test Planning - XML Schema

Test Execution - Testing in the Cloud

WebServices (WCF)

<< Realtim

e Test Results >>

<< Realtim

e Error Reporting >>

<< Check Test Schedule >>

Test Execution - Automation Scheduler

Selectable list of user case stories (Test Sets)

Available machines

Global Stop Buttons (Stops all tests on all running machines)

Global Pause button

Global Play button

Web Services (WCF)

XML

Communication - Message Definition (WCF)

ANZTB 2010 – ConferenceHybrid Keyword Data Driven Automation Frameworks by Jonathon Wright

Tuesday, 2nd March 2010

© ANZTB 2010

Windows Service (Lifecycle) Event / Object Handling

Supported events:

� button.click

� text.parmsetcheck

� text.validate

� link.click

� reporttype.selecttreeitem

� reporttype.opentreenode

� list.parmsetcheck

� checkbox.select

� tree.validate

� tree.parmsetcheck

� radio.parmsetcheck

� checkbox.parmsetcheck

� checkbox.validate

Business Level Keywords (Process)

Business Level Keywords represent a number of logical test steps that make up a business process i.e. “Login.Process”

User Stories are represented by a number of Business Level Keywords:

Each User Story represents client Market Configuration Testing mappings:

LoginFrm.Logo.Image

LoginFrm.UserName.Label

LoginFrm.Password.Label

LoginFrm.LoginBoxHeader.Label

LoginFrm.LoginButton.Label

LoginFrm.ChangePWButton.Label

LoginFrm.ForgotPWLink.Label

LoginFrm.ServiceDeskTel.Label

LoginFrm.ServiceDeskMail.Label

LoginFrm.Copyright.Label

ABCD

E

FGHIJ

F

G

H I J

A

B

C

D

E

Market Configuration Testing (Process)

Process the first test step configuration data value located in the “Test Repository” i.e. #dh.MyServices_Login_UserNameLabel

Using the data value to confirm that the “Market Requirement” exists for the target market (<Client> + <WorkStream>)

Once confirmed the configuration value can be extracted from the “Market Specification” by selecting the target market (<Language>)

Test Step 1 – Using the “Configuration” value

to verify that market configuration is correct:

UserNameLabel = “User Name:”Result = FAIL

Object Recognition (Smart Identification)

Get the test step object name from the Test Repository (MasterDriver) i.e. LoginFrm.Password.Input

Using the object name extract the object class & associated value from the Object Repository (MasterParameters) i.e. Html.EditBox & “password”

Using the object class extract the primary & secondary class tag identifiers from the Class Definition (MasterParameters) i.e. “InnerText” & “Type”

Attempt #1 – Using the “WebElement” Class

Describe the Object Physical Description:

WebElement Object= [@InnerText = “password”]

Attempt #2 – Using the “InputTag” Class

Describe the Object Physical Description:

InputTag Object= [@Type = “password”]Physical Object

Physical Description

Object Recognition (Fuzzy Logic)

FlexObject flexObj = @"/dom[@domain='']/frameset/frame/body/flexobject[@id='CrossShop']";FlexObject flexObj = @"/dom[@domain='']/frameset/frame/body/flexobject[@id='CrossShop']";

IList<Ranorex.Button> flexButtons = flexObj.FindDescendants<Ranorex.Button>();IList<Ranorex.Button> flexButtons = flexObj.FindDescendants<Ranorex.Button>();

Get all the screen objects on the flex page.

Retrieve all the Buttons on the page.

For ( int I = 1; >= flexbuttons.Count; i++)

{

// get the xpath for each element.

RxPath me = flexButtons.ElementAt<Ranorex.Button>(i).GetPath();

// convert the xpath to a string

string[] str = me.ToString().split(‘/’);

// convert xpath to a split of strings

string[] str1 = xPath.Split(‘/’);

// compare both string paths until a common tree route is found (this will be your button)

CompareTreePaths( str, str1 )

}

When you have identified a tree path, you can then use it in the way described in the MasterDriver file (e.g. flexButton.Click).

ANZTB 2010 – ConferenceHybrid Keyword Data Driven Automation Frameworks by Jonathon Wright

Tuesday, 2nd March 2010

© ANZTB 2010

Unified Test Reporting (UTR) Unified Test Reporting – Real-time Log

TestRunnerInput (XML) Output (WebService)

.NET Object WCF

Error/Event Handling

Logging (Debug)

Results (XML)

Process Improvement

� Agile Planning – Team Foundation Server linked to Task Board (SfTS v2);

� Coding Standards – Code reviews (internal/external to team), coding tools, unit test standards, agile hints, logging etc etc;

� Testing - Internal peer reviews, unit tests developed, exploratory testing for each story as a task, Integration test story in each sprint;

� Wiki – Knowledge capture including online framework documentation (screenshot)

� Technical Notes /Debt - on project wiki (intranet SharePoint).

Key Achievements

• Centralised - management of test assets to provide open connectivity as a generic source of test scenario data;

• Dynamic - generation of sanitised input test data validated against business rules along with an object repository based on a combination of descriptive programming containing XPATH regular expressions;

• Legacy - support for static data types plus full/partial fall back support for manual test execution;

• Unified reporting - using funnel virtualisation techniques to analyse the quality of builds, assisted by diagnostic data adapters;

• Generic reusable - test components, modules and libraries providedreliable processing of the entire test automation effort.

SSUMMARYUMMARY

•• HybridHybrid –– utilising the best technologies and resources to do the job;utilising the best technologies and resources to do the job;

•• Keyword Keyword –– creating simple and robust test scenarios written in business creating simple and robust test scenarios written in business

level keywords combined with natural language;level keywords combined with natural language;

•• Data Data –– effective use of dynamic business data to provide an input sourceffective use of dynamic business data to provide an input sourcee

•• Driven Driven –– processes the test component actions, objects and events processes the test component actions, objects and events

seamlessly;seamlessly;

•• Automation Automation –– that is collaborative, distributed and scalable;that is collaborative, distributed and scalable;

•• Frameworks Frameworks –– independent of application or environment under test;independent of application or environment under test;