© TRASYS 2008

TRASYS TAAS “Testing As A Service”

November 2013

© TRASYS 2008

Introduction

Testing As A Service: Definition

TAAS is a model of software testing whereby TRASYS undertakes

the activity of testing applications & solutions for our customers as a

service with following traits:

On-demand services

service delivery function & service governance

Well-defined & repeatable services (service catalogue, RACI, entry & exit

criteria, )

Outsouring using a shared service centre

Pay per use (based on test effort estimation techniques)

Well defined test process & methodology

Professional / certified test resources

Supported by (cloud based and virtualised) test tools & test environments

Testing as a Service: Key components

Managed Services

With KPIs and SLAs

Centralised

Resource allocation

&

global X-shore

capability

Defined list of services

and test activities

Remote, global

and secure

Acces to AUT and

test environments

Test tools

as a service

(SAAS Cloud)

Certified and

professional testers

Unit price for

each service

(test case, use case ,

function point, LOC, etc)

Test automation

and simulators

enhance

productivity

On demand

and virtualised

test environments

Global Network of Test

Centres Of Excellence

Testing as a

Service

Testing As A Service: Key Benefits

Testing As a Service

Benefits

Reduce TCO - cost of quality (between 25% and 35%)

Optimize governance &

streamline communication

Access to ‘niche’ expertise Predict service cost (fixed price)

Shift from CAPEX to OPEX (pay for what you use)

Schedule acceleration & flexibility (Rapid provisioning – shared HR and IT

resources)

Predict service outcome (result commitment – SLA’s – defined services)

Reduce risk Levels & improve

product & service quality

© TRASYS 2008

Testing As A Service Service Delivery Model and assets

Testing as a Service: Our assets

ISO 9001:2008 certified Test Centre of Excellence

8 Western European locations and Ukraine

ISTQB, Prince2 and Agile certified testers

80% certified

Resource pool > 100

Majority of testers are senior or expert level

( between 5 and 10 years experience)

Test tools & test environments offered as a service

Secure remote access

Cloud based

Virtualized and on demand

See annex 1 for more details

Testing as a Service: Service Delivery Model

Resource

allocation

Mgr.

Release

Test Centre Of Excellence

SAAS test mgt.

& defect mgt.

tools

Virtualised On Demand

test environments

Service catalogue

Test Mangement

& goveranance

Nearshore team

TRASYS Service Delivery Manager

Testing best practices &

metholodogy

Client Service Manager

Fixed Price

Service

Contract

SLAs and

metrics

A Projects A Applications

Simulators &

test data mgt.

tools

On Demand Request

Service Delivery

Offshore team

Releases

On site team

Testing as a Service: Our vision

• Business & technology risks define test

strategy

• Traceability matrix

• Requirements coverage analysis

Risk & Requirements Based

• Test management & defect management

• Test automation, test data analysis & mgt.

• Performance testing & profiling

• Security , penetration & vulnerability

• Code review and inspection

Tool supported

• Focus on (important) business processes

• Black box testers need to master the business

• TCO & ROI drive test strategy & master plan

Business driven

• Based on best practices:

ISTQB , ISO 9126, ISO 29119

• Test design techniques for test cases & test data

• Test effort estimation techniques

• (Master) test plan & formal test strategy

Structured

Test strategy, tools & checklists (= framework) for:

• Mobile applications

• SAP, CRM & ERP

• Cloud SAAS

• Webservices & SOA

• Mainframe <> J2EE <> RIA & HTML5 <> …

Technology specific

• Test activities should be embedded in the SDLC

• Testing starts at inception of a project / release

• Testing should be adapted to the lifecycle:

Agile, iterative, V-model

Early & Adapted to lifecycle

© TRASYS 2008

Testing As A Service Service Catalgue

Testing as a Service: Service Catalogue

Data driven I.M

test automation

Code review &

requirements review Cloud testing

Keyword driven test

automation Portability testing

Mobile application

testing

Agile test driven Security & reliability Big data - DWH and

BI testing

Model based

test automation

Performance testing

& profiling CRM and ERP

Test automation Non functional testing Functional & business

oriented Test solutions

Integration and SOA

testing

E2E business

process testing

Usability & Crowd

testing

Functional system

testing

© TRASYS 2008

Testing As A Service Test Solutions

Testing as a Service: Test solutions

Data driven I.M

test automation

Code review &

requirements review

Keyword driven test

automation Portability testing

Cloud testing

Agile test driven Security & reliability

Mobile application

testing

Model based

test automation

Performance testing

& profiling

Test automation Non functional testing

CRM and ERP

Functional & business

oriented Test solutions

Integration and SOA

testing

E2E business

process testing

Usability & Crowd

testing

Functional system

testing

Big data - DWH and

BI testing

Testing as a Service: CRM & ERP

Testing of a ‘commercial off-the-shelf’ database software application(s) supporting

customer relationship management and enterprise resource planning business

processes.

Some leading providers of CRM / ERP

Definition

• Succesfull selection and Integration of the ‘package’ leading to ‘acceptance’

• (Test) Data accuracy, data quality and security during storage and retrieval

• Understand the (impacted) business process(es)

• Understand the test object ( platform) , the changes and their impact

Key

challenges

• SAP solution manager, SAP TAO, SAP ECATT, SAP TDMS

• Predefined test (automation) libraries & accelerators for SAP, Peoplesoft & Oracle (order to

cash, procude to pay, etc)

• HP QTP

TRASYS

tools

• (automated) Data migration & platform migration testing

• Customisation, parameterisation and usability testing

• Assistance during evaluation / testing of prototypes - packages

• (automated) version upgrade (regression) testing

• Installability, configuration & deployment testing

• E2E business process testing & system integration testing

• (automated & data driven) functional acceptance testing and testing of reports

• Application Lifecycle Test Management services ( ALM)

TRASYS

Services

Testing as a Service: Cloud Testing

Testing conducted on the applications and services provided via cloud computing:

‘test object’ i.e application under test and / or testing services themselves are

provided via cloud computing using PAAS / SAAS and APIs

Defintion

• Performance, scalability, capacity and availability

• Access, privacy & security

• Migration, interoperability & reversibility (avoid “vendor lock in”)

Key

challenges

• HP Loadrunner

• SOASTA Cloud test

• SoapUI

TRASYS

tools

• Load balance and stress testing for a range of simulated usage conditions

• Role based & tool based security testing

• Cloud migration testing (data migration, repository migration)

• E2E system integration testing ( Hybrid cloud, interfaces between Cloud SAAS

and back end systems and databases, API testing, SOA testing)

TRASYS

Services

Testing as a Service: Mobile Application Testing

The functional and non functional testing of mobile-based native (or hybrid)

applications.

Some leading mobile app. platform providers: SAP, IBM, Kony, Antenna, Jquery mobile,

Adobe, Appcelrator

Definition

• Understand the native application platform (hardware, OS) and network types

• Testing needs to ascertain that the app can be successfully downloaded to the device,

executed on the device and interact with the supporting backend & infrastructure and

updates can be made (push out)

• Understand the non functional requirements, risks & ‘vulnerabilities’ (security, performance,

usability, portability…) related to mobile application development and (end user) usage

Key

challenges

• Predefined mobile application checklists and ‘standard’ test cases (MAT starters kit)

• Our mobile application test lab contains multiple fysical devices & emulators in order to

support different test objectives. This is also supported by remote cloud based devices .

• We can test multiple networks from our different locations (i.e GPRS tunneling effect)

• Automated mobile test execution using emulators and test automation tools (see annex 1)

TRASYS

tools &

assets

• Risk based multidevice and multi OS testing, including backward compatibility

• App lifecycle mgt testing ( from store readyness to push out of upgrade app versions)

• Functional testing , security testing, performance testing, E2E business process testing,

back end integration testing and usability testing for mobile applications

TRASYS

Services

Testing as a Service: Big Data, DWH and BI

Testing of the data warehouse i.e the main organisations ’repository

of transformed and cleaned historical data’ and business intelligence and

decision support applications

Some leading providers:

Defintion

• Availability of high quality (test) data in order to cover requirements (i.e reports or queries

and ETLs)

• High volatility of business rules & requirements in DWH/BI testing

• Sensitivity, complexity & confidentiality of (test) data

Key

challenges

• Tricentis Tosca Testsuite (for DWH testing)

• Analytical simulation engine (accelerator)

• Data driven testing tools and model based testing tools

(See next slide)

TRASYS

tools

• DWH, DI and Big data testing:

• Source data extraction testing

• ETL process validation (including statistical & dimensional analysis and data quality

validation)

• E2E integration testing of ETL batch jobs

• Report validation, Cube validation, dashboard validation

• Non functional testing (performance , security)

• (Test) data mining, combined with ‘synthetic’ test data creation

• Simulation of business rules while processing historical and/or near-to real time transactions

in order to test systems

TRASYS

Services

© TRASYS 2008

Testing As A Service Focus on specific services

Supporting test services

Testing as a Service: Supporting test services

Data driven I.M

test automation

Code review &

requirements review

Keyword driven test

automation Portability testing

Cloud testing

Agile test driven Security & reliability

Mobile application

testing

Model based

test automation

Performance testing

& profiling

Test automation Non functional testing

CRM and ERP

Functional & business

oriented Test solutions

Integration and SOA

testing

E2E business

process testing

Usability & Crowd

testing

Functional system

testing

Big data - DWH and

BI testing

Testing as a Service: Model Based Testing (MBT) & Data Driven Testing (DDT)

Testing based on a model of the component or system under test, e.g., reliability growth models, usage

models such as operational profiles or behavioral models such as decision table or state transition

diagram.Test cases are derived from a model such as finite state machine , UML, state chart, Markov

chain that describes (mostly functional) aspects of the application under test

-> Model based testing is supported by data driven test automation (DDT)

-> Model based testing is a way of simulation early in the SDLC

Defintion

• Cost reduction

• Increased test coverage

• early validation of design model

Key

advantages

• Test Optimal (MBT tool)

• MaTeLo (MBT tool)

• HP QTP (DDT automation tool)

• Selenium ( test automation tool)

TRASYS

tools

• Test data & test case generation using

design techniques (boundary values, pair wise,

branch, decision, transition, state, requirements,..)

• Model based & data driven test automation

• Test data management

TRASYS

Services

Testing as a Service: Security Testing

Defintion

Key

challenges

TRASYS

tools

• Vulnerability scanning & assessment

• Penetration testing / attack based testing (= systematic probing of systems with the goal

of gaining access to the information they contain)

-> Open Web Application Security Project (OWASP) top 10

most critical web application flaws

-> Open Source Security Testing Methodology (OSSTM)

TRASYS

Services

• Acunetix

• Webscarab

• IBM Appscan

• See appendix 1 for full list

Determine that an information system protects data and prevents unauthorized

access, whether accidental or deliberate, to programs and data.

The six basic security concepts that need to be covered by security testing are: • confidentiality

• Integrity

• identification & authentication

• Availability

• Authorization

• non-repudiation

Testing as a Service: Usability & Crowd Testing

Defintion

Key

challenges

TRASYS

tools

• Requirements based usability testing, including

prototype & mock up validations during inception

of SDLC using for example ISO 9241

• Real-world usage validation in order to

continuously improve the systems * Use of production data analytics & usability tools

* Crowd Testing = crowd sourcing + cloud test platform

TRASYS

Services

• Chalkmark, Clickheat

• Google analytics

• Usertesting.com, Userzoom

• Browser cam (cloud based)

• UserReplay

• Avoid that end users / customers leave (abandon) when a system is difficult to use, a

homepage fails to clearly deliver information i.e E-commerce: if users cannot find the

product, they cannot buy it either ( impact on revenue)

• The functionality of our application is state of the art, but users lose time trying to figure out

how to operate it ( impact on productivity)

Testing the ease with which the user interfaces can be used and whether the application or the

product built is user-friendly or not.

Usability testing also reveals whether users feel comfortable with your application or Web site

according to different parameters - the flow, navigation and layout, speed and content -

especially in comparison to prior or similar applications.

Testing as a Service: Performance Testing

Defintion

Key

challenges

TRASYS

tools

• Load testing

• Stress testing

• Scalability testing & capacity planning support

• Code review with performance focus – performance profiling

• Reliability testing

TRASYS

Services

• Loadrunner

• Jmeter

• Jprofiler

• Several code review & inspection tools ( see Annex 1)

• Avoid that end users / customers leave (abandon) when a system is ‘too slow’

to use, a homepage fails to deliver information with an acceptable response

time -> impact on revenue

• The functionality of our application is state of the art, but (internal or external)

users lose time to operate and use the functionality -> impact on productivity

• System is not available due to overload -> impact on revenue

• System is oversized or undersized due to lack of performance testing ->

impact on Capex / Opex

The process of testing to determine the performance of a software product.

© TRASYS 2008

Testing As A Service Implementation approach

Testing as a Service: Implementation approach

Critical Success Factors

Defined as major business

transformation & major

change IN YOUR

ORGANISATION

Managed as a project with

a plan, budget , resources,

scope, dependencies,

risks,…

Business case

supported by share -

and stakeholders

Service contract with

RACI and defined service

inputs and outputs

Collaboration model and

governance model

Testing as a Service: Implementation approach

Prepare phase

•Feasibility study – Proof of Concept – Business Case

•Test maturity assessment

• Information and communication (generate awareness)

Service Design phase

•Target operating model & test process – governance model – collaboration model - RACI

•Service contract – document of understanding (scope of services & projects / assets – pricing – planning)

•Transition , change and deployment plan (including training and internal communication)

Implementation phase

•Knowledge transfer – information transfer – access to test object , test environments & tools

•Proof of concepts

•Mass deployment

Operations phase

•Run operations as a service

•Monitor and report on operations (KPIs, activity reporting)

•Manage demands and service delivery

Improvement phase

•Operational excellence through continuous test process improvement and service improvement

Annex 1 : Test tools

Defect

management

Test

Management

Test

automation

Performance

testing &

profiling

Code

review

&

inspection

Usability

and

prototype

testing

Security

testing

Agile

testing

Mobile

application &

cloud testing

SAP

testing

HP Quality

Centre - ALM

HP Quality

Centre - ALM

HP QTP HP Loadrunner FindBugs

Chalkmark Acunetix

WVS

Jira

Greenhopper

Fonemonkey SAP TAO

Jira Spiratest Selenium Jmeter

PMD

Clickheat

BackTrack

SmartBear

ALM

Perfectomobile

(cloud based)

SAP

Solution

Manager

Bugzilla Flexmonkey Jprofiler

CheckStyle

Google

analytics Paros Proxy

MS Visual

Studio and

TFS

IBM Worklight

(mobile) and IBM

Smartcloud

SAP

eCATT

Mantis SOASTA

JavaNCSS

Usertesting.

com Tamper

Data

Mylyn for

eclipse

SOASTA

(cloud based)

• CloudTest

• TouchTest

HP QTP

SAP plug

in

SOAP UI

(SOA testing) Cobertura

Userzoom

WebScarab

HP Agile

Manager

(cloud

based)

Open Cloud

architecture

( Cloud Foundry)

Test Optimal

(model

based

testing)

SonarQube

Google

Website

Optimizer

CSRF

Tester

W3C mobile ok

checker

MaTeLo

(model

based

testing)

Browser

cam (cloud

based)

IBM

rational

AppScan

Deviceanywhere

(cloud based)

UserReplay Xamarin

(cloud based)

Thank you for your attention!

28 TRASYS Confidential