Top100summit christina

Automation Test Framework Continue Improvement

Welcome PresentationPresented by Christina Geng

AGENDA

❶ Case Introduction❷ Testing Strategy Evolution❸ Case ROI Analysis❹ Success factor❺ Revelation

Title ：7 Times Evolution of Test Automation Framework under

JS Product for Efficiency Improvement

Target ：

❶ Build a robust test automation framework

❷ Manage test cases efficiently

❸ Reduce test case running time and maintenance cost

❹ Shared framework under scrum development

CASE INTRODUCTION

WHO AND WHAT?

Data Analysis Visualization

• reporting, online analytical processing • analytics• data mining• process mining• event processing• business performance management• benchmarking• text mining• predictive analytics

WHO AND WHAT?

Data Analysis Visualization

• reporting, online analytical processing • analytics• data mining• process mining• event processing• business performance management• benchmarking• text mining• predictive analytics

Programming language: Javascript/Java

Provide service: The Visualization Service is a software module which provides various type of visualization objects that can be

used in different applications on different platforms (Web, Mobile and Desktop).

Visualization types: • A basic visual component like chart (Bar, Column Pie, Line, etc ), Geo Map, Gauge, Table or Grid.

• An advanced visual component like (Tree map, Candlestick, Tagcloud) • Above visual component with interactions and animations

• Combination of multiple visual components, it could be composed by same type of visual components or different type of component.

AGENDA

❶ Case Introduction❷ Testing Strategy Evolution❸ Case ROI Analysis❹ Success factor❺ Revelation

We have it!

Selenium

Data D

riven

Jenkins

Test Link

Full Automation

Developer environm

ent

What’s m

ore…2011.6 2011.12 2012.032012.06 2012.09 2012.10 2012.12

VERSION-1

Test page

SilkTest

Pie Bar Line BackgroundTitle Legend Layout Tooltip Theme ValueAxis AnimationPie y y y y y yBar y y y y y y y y yLine y y y y y y y y y

Scheduler: ASTECTC Management: ExcelTC: test pageTest tool: SilkTest

VERSION-1

Test page

SilkTest

+ -HTMLTest

page

1. First automation framework setup2. Overview functional test case

coverage3. Reduce testing overlap for shared

component

writing test case

maintenance

the relation between TC and record

resource

We have it!

Selenium

Data D

riven

Jenkins

Test Link

Full Automation

Developer environm

ent

What’s m

ore…2011.6 2011.12 2012.032012.06 2012.09 2012.10 2012.12

VERSION-2

DIV

DIVDIV

DIVTest page

Selenium

Scheduler: ASTECTC Management: Excel/XML (test manager)TC: test pageTest tool: Selenium Test Driver

Other tools• Auto generate test plan from Excel to xml• Test Manager

DIV

DIVDIV

DIVTest page

VERSION-2

+ -1. Reduce test pages2. Reduce resource running cost3. Test steps turns to be keyword driven in xml4. Easy case migration5. Write test case in .xls, auto generate .xml

Selenium

DIV

DIVDIV

DIVTest page

test driver

test case sync

case unit

test data redundancy

We have it!

Selenium

Data D

riven

Jenkins

Test Link

Full Automation

Developer environm

ent

What’s m

ore…2011.6 2011.12 2012.032012.06 2012.09 2012.10 2012.12

VERSION-3

Wrapper API

Unified Test Driver

Abstract Test Data

Discard Test Page

And How?

VERSION-3

Wrapper API

backgroudcolor

visiblelegendposition

verticalOffsethorizontalOffset

visiblelabelFont

labelColorformatString

layoutpaddingtooltip

enablecustomization

textmainValue

subValuetitle

main subSelectability

modedeSelectale

defaultSelectedItemsxAxisMemberyAxisMember

barbarMarkerSize

measureMember

initChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);

updateChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);

testUtil.generateChart(viz/line; TestData.a1a1m1m1)

testUtil.generateChart(viz/line; TestData.a1a1m1m1; ['legend.visible',false])

VERSION-3

Abstract Test Data

publicpublic

privateprivate

testUtil.generateChart(viz/line; TestData.a1a1m1m1; ['legend.visible',false])

mySQL

testUtil.generateChart(viz/line; TestData.a1a1m1m1; legendVisibleTrue)

VERSION-3

Discard Test Page

<testsetcase id="test.workflow.layout.line.showTitle" enabledtype="enabled"/><testsetcase id="test.workflow.layout.line.hideLegend" enabledtype="enabled"/><testsetcase id="test.workflow.layout.line.hideAxis" enabledtype="enabled"/><testsetcase id="test.workflow.layout.line.showAxisTitle" enabledtype="enabled"/>

titleVisibleTrue = [['title.visible',true]];titleVisibleFalse = [['title.visible',false]];titleNormal = [['title.visible',true],['title.text',"Set customer title"]];titleLong = [['title.visible',true],['title.text',testData_Text.Long]];titleAlignmentCenter = [['title.visible',true],['title.alignment',"center"]];titleAlignmentLeft = [['title.visible',true],['title.alignment',"left"]];titleAlignmentRight = [['title.visible',true],['title.alignment',"right"]];legendVisibleTrue = [['legend.visible',true]];legendVisibleFalse = [['legend.visible',false]];legendHierarchicalTrue = [['legend.visible',true], ['legend.isHierarchical', true]];legendPositionLeft = [['legend.visible',true], ['legend.position',"left"]];legendPositionRight = [['legend.visible',true], ['legend.position',"right"]];legendPositionTop = [['legend.visible',true], ['legend.position',"top"]];legendPositionBottom = [['legend.visible',true], ['legend.position',"bottom"]];

VERSION-3

Unified Test Driver

selenium1

selenium2{StartBrowser}IE, Chrome, Firefox (selenium2)

RemoteWebDriver webDriver = null;

webDriver = new FirefoxDriver();

Robot()

ImageComparer

VERSION-3

Unified Test Driver

selenium1

selenium2

{StartBrowser}Safari (selenium1)

SeleniumServer seleniumServer = new SeleniumServer();

seleniumServer.boot();

selenium = new DefaultSelenium("localhost", 4444, "*safariproxy "+ safariLocation,urlRoot);

CommandExecutor executor = new SeleneseCommandExecutor(selenium);

DesiredCapabilities dc = new DesiredCapabilities();

webDriver = new RemoteWebDriver(executor, dc);Robot()

ImageComparer

VERSION-3

Unified Test Driver

selenium1

selenium2 {TakeSnapshot}Point p = this.getElementOffset(targetElement);

Dimension d = this.getElementSize(targetElement);

BufferedImage bi = new Robot().createScreenCapture(p.x, p.y, d.width, d.height);

Robot()

ImageComparer

VERSION-3

Unified Test Driver

selenium1

selenium2

ImageComparer

{FileCompare}cmd.exe ImageComparer.exe –file filepath1 filepath2 – threshhold 100

Robot()

We have it!

Selenium

Data D

riven

Headless

Test Link

Full Automation

Developer environm

ent

What’s m

ore…2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12

VERSION-4

Tooltip Selection

Gesture

CSSButton

Link

Dropdown

Checkbox

Table Text

Scroll

Hover

Animation

Theme

Upload

Download

Tooltip Selection

Gesture

VERSION-4

Selection

VERSION-4

Gesture

VERSION-4

var eventObj = {type : originalEvent.type,timeStamp : originalEvent.timeStamp,pageX : originalEvent.pageX,pageY : originalEvent.pageY

};eventRecord.push(eventObj);

var handler = function(event){var clickEvt = document.createEvent('MouseEvents');clickEvt.initMouseEvent(event.type, event.pageX,

event.pageY…);d3.select('.main .plot')[0]

[0].dispatchEvent(clickEvt)};testAPI_lasso_heatmap = [

{"type":"mouseover","timeStamp":1348739963751,"pageX":126,"pageY":94},{"type":"mouseover","timeStamp":1348739963751,"pageX":485,"pageY":94},{"type":"mousemove","timeStamp":1348739963766,"pageX":485,"pageY":208},{"type":"mousemove","timeStamp":1348739963773,"pageX":126,"pageY":208}

]

VERSION-4

Record

Replay

VERSION-4

start to checkdiff image folderdiff image location

p4 folder

replace BM directly

We have it!

Selenium

Data D

riven

Headless

Test Link, Jenkins

Full Automation

Developer environm

ent

What’s m

ore…2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12

VERSION-5

How to arrange test case?

VERSION-5

Test Link

TC (.xls/.xml)

Executable TP

import

export

Result logsave

VERSION-5

Continue Integration

1. Trigger

CI Server Test Executor

Build Machine Code repository

Manual Auto

TC mngmt

Test Case Pool

Test Driver

Selenium2 Java/C++

Web Server

2.TP/Platform/build info

resources

4. get assets

libs

4. get test plan3. Build info 5. result write back

5. result log

.xml.xls

VERSION-5

Where we are.

1. Trigger


Build Machine Code repository

Manual Auto

TC mngmt

Test Case Pool

Test Driver

Selenium2 Java/C++

Web Server

2.TP/Platform/build info

resources

4. get assets

libs

4. get test plan3. Build info 5. result write back

5. result log

.xml.xls

VERSION-5

Test Link

JenkinsJava Tool

PerforceWhere we are.

We have it!

Selenium

Data D

riven

Headless

Test Link, Jenkins

Full Automation

Developer environm

ent

What’s m

ore…2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12

Engineering

VERSION-6

http://casperjs.org/

VERSION-6

Paint

Layout Display

http://phantomjs.orgPhantomJS (Headless Webkit)

VERSION-6

Test driver inner logic

CasperJS IE/Safari/FF/Chrome

configBrowser?

Open browser

RunJS (test step)

….

Take Snapshot

Result validation

Save result log

Read JS listTP

PhantomJS:1. open webkit2. runJS3. Take snapshot4. close

not completenot complete

Reporting

And that’s OK?

We have it!

Selenium

Data D

riven

Headless

Test Link, Jenkins

Full Automation

Developer environm

ent

What’s m

ore…2011.6 2011.12 2012.03 2012.06 2012.09 2012.10 2012.12

trigger


Build MachineP4 server

TC Management

Test Case Pool

Test Driver

Selenium2 Java/C++

get resources

sync up

.java

per check-in

local

Test Executor

result log

VERSION-7

Casper/PhantomJS

trigger


Build MachineP4 server

TC Management

Test Case Pool

Test Driver

Selenium2 Java/C++

get resources

sync up

.java

per check-in

local

Test Executor

result log

VERSION-7

Casper/PhantomJS

Eclipse Plug-in

PerforceJenkins

1. Set up environment

2. Test case execution

3. Test result reporting

4. Maintenance

CASE ROI ANALYSIS

1) Test machine2) Assets (util, BM…) 3) TP4) Libs

1) Browser compatibility2) Run in headless

1) Email sending2) Sonar integrated with CI

1) BM update2) TC management

Functional Test: BAT (nightly), FA

1000+ cases

4.5h47min

SUCCESS FACTOR ❶ ❷❸❹

Focus


Team work


Continue Improvement


Learn 360°

OUR REVELATION

不被传统的测试方法束缚；从产品用到的技术找到突破；持续改进的思想；不断尝试的理念

CONTRIBUTORXiaolu, Ye

Christina, GengHelen, JiangChen, ZhouPeipei, Jiang

Coco, DaiGrace, ZhaoMinghao, LuKent, Wang Izzie, Wang

Phil, Xu

SUPPORTED FROMDaoyu, Wei

Li, ZhaoJimmy, YangAlex, FengAlex, Su Jiwei, Li ... …

THE ENDNO

Documents

Top100summit christina