25 May, 2017

Sudeepta Guchhait

Test Paradigms:- Applying test integrity under challenges

2

Schema

3

Exchange - Big Rock Implementation

Mega test project

Unique challenges

Our journey – approach towards continuous testing

Critical success factor

Special about project

Guide you through the challenges faced during test life cycle and how to master the mitigation approach. Speak about how to succeed in continuous test improvement

Exchange - Big Rock Implementation

4

… to stay in the game, we needed a single multi-asset platform with new features to support end-to-end workflows of derivatives communities

Then culture - Why were we upgrading? Applications, Technology & Hardware end of life. Lower performance, lesser interoperability, Lack of efficiency & interoperability btw platforms

Adoption - Changes brought! New platform for Listed derivatives, OTC derivatives. Integrated Clearing & collateral mgt systems Physical gateways 2 new low latency protocols Operation & process improvements

Derivatives Platform OTC

FIX 5.0 GUI OMNet OUCH GUI

Interfaces

Functionalities

Orders

Quotes

Trade Reports

Market Data

Risk Controls

Clearing & CMS

OMNet API

OUCH

ITCH

Pre Trade Risk Control (API + GUI)

OMNet API

FIX 5.0 API

OTC GUI

Trade Reports / Market Data / Risk Controls

Clearing Workstation

Other Benefits

5

+ Embedded modern tools to improve delivery efficiency at Xchange.

+ 22 hours trading longest trading hours in Asia (6:45AM – 4:45AM)

+ Improved co-location culture.

+ Lower latency and higher throughput with New application

+ A reduction in operational costs and headcount for BAU activities

+ Processes that enable frequent releases with higher confidence.

What is a Mega - testing project?

6

Scale

Size

Scope

• Complex business

• Business impact

• Critical enterprise applications

• Multi application vendors

• Massive budget

• Large team

• Multiple test environments

• High number of applications/downstream

• Usage of many test tools

• Extensive business functions

• Millions of transactions

• Business process

• User stories (requirements)

• Hybrid SDLC

What Business Demands?

7

Strive towards “0” prod incidents

Waterfall + Agile SDLC

Driving automation

Continuous Integration

Project outcomes & benefits

BUSINESS FACING

TECHNOLOGY FACING

MEM

BER

FA

CIN

G

Trading & Clearing ITCH/OUCH protocols Glimpse for data vendors Trade handler Risk Groups

Collateral Management Risk Management External system

Reporting Dashboard Heat maps

Market data Data Analytics Algo System Central Data source Market Surveillance

Unique Challenges

8

Platform transformation + multiple applications

Cultural issues

Multiple application vendors

Hybrid SDLC

Imposing different types of test

Testing – “Design for failures”

End to end application test is difficult to follow

Reliability & Stability of application

Availability/Maintenance of test infrastructure

Not enough time – (tight deadline)

Managing complexities - (missing requirements, embracing CRs)

Pressure to deliver on time - (KPI project, reputational risk)

Cross integration test – (application + data + diversified api)

Our Journey – “Mega testing” – Are you ready?

9

Application Delivery Challenges

10

Design & Development

Constrained Environments

Less time, tool time

Manual testing

Too many disparate tools

Automation (F /NF)

Production Roll Out

Waiting time Delays Delays Delays

Build Test/Deploy Implement Test/Deploy Test/Deploy

Developers Testers Release Team Prod Support /

Operations

Code | Integrate | Build Functional / API / Automation

Automation + NF testing Deploy to Prod defects defects defects

Re-execute Re-execute Re-execute Re-execute

SLOW & INEFFICIENT EXCESSIVE IDLE TIME, DELAYS AND LOT OF REWORK

Hurdles We Had To Overcome

11

Manual testing was comfortable to everyone

Hybrid life cycle

Downstream test automation

API / FIX protocol testing

Skill transformation & Capability uplift

Data Migration testing

NF testing (Perf, Capacity, Failover & Latency)

One business day simulation

Critical path/Pattern testing

Embraced Test Harness (Risk based & Combinatorial test)

Bi-Directional traceability

Timeline – Improvement Journey

Re-defining strategy 1. Standardise test process 2. Identify quick wins 3. Define strategic

solution 4. Define tactical solution

Launch Derivatives Platform.

2015 / Q3

2016 / Q1

2016/ Q2

2016 / Q3

2016 / Q4

12

Building Champions 1. Cross training 2. Developing automation

suite 3. Driving sense of

competition 4. Development of

platform focused tactical automation

5. Realise benefits from early automation

6. Engage members

Platform test 1. Automated test (API) 2. User Acceptance test 3. Automated downstream 4. Non functional test 5. Drive upstream testing

Continuous improvement 1. Infuse DevOps 2. One business day simulation 3. Behaviour driven testing 4. Industry wide testing 5. Member test

Business Impact Driven Test Approach

Assess the business impact of each activity Assess the probability for failure Determine the risk based on impact and probability

Application 1

Main Functionality 1.1

Function 1.1.1

Requirement

Requirement

Requirement

Requirement

Requirement

Requirement

Function 1.1.2

Main Functionality 1.2

Function 1.2.1

Requirement

Structuring Test Requirements

Determining the Risk

Analysing Business Impact

Analysing Probability to Fail

13

Criteria \ Results A

High Risk B

Medium Risk C

Low Risk

Types of Transaction / Algo

Calculation/Validation

Change of assignment

Display

Business impact Legal Wrong

information None

Frequency of usage Very Often Often Rare

Number of members affected

Platinum Gold Few

Criteria \ Results 1

Most likely 2

Possible 3

Unlikely

Change in API New

messages Changed message

Unchanged

Functional impact Upstream Down

Stream Support

Impact to member Critical Manageable Internal

HIGH

MEDIUM LOW

Impact \ Probability III

Unlikely II

Possible I

Most Likely

Impact to Business Least Recovery Financial

Impact to Reputation Least Manageable Severe

MAS Reportable None May be Yes

Tactical Solution

Application 1

Main Functionality 1.1

Function 1.1.1

Requirement

Requirement

Requirement

Requirement

Requirement

Requirement

Function 1.1.2

Main Functionality 1.2

Function 1.2.1

Requirement

Functional Complexity Determines Test Effort

Analyse the functional complexity together with technical architect Determine test effort by combining risk & complexity

Structuring Test Requirements Analysing Functional Impact

Analysing Probability

14

Criteria \ Complexity I

High Complex

II Medium Complex

III Low

Complex

Number of Functional application pretentious

> 11 6 – 11 < 6

Number of Calculus application affected

> 04 2 – 4 Just 1

Number of Dashboard affected > 05 1 – 5 None

Impact \ Complexity I

High Complex

II Medium Complex

III Low

Complex

Critical 10, 5 8, 2 6, 3

Important 6, 9 6, 3 5, 6

Nice to have 4, 2 3, 5 2, 8

Tactical Solution

The Risk Determines the Test Approach

15

Technique: Systematic testing using critical component testing, API messages and root-cause analysis

Approach: Automated: 30% Manual: 70%

Tactical Solution

HIGH Risk

A MEDIUM

Risk B

LOW Risk

B

Technique: Systematic testing with patterned & combinatorial data points testing, and root-cause analysis

Approach: Automated: 20% Manual: 80%

Technique: Systematic or ad hoc testing

Approach: Automated: 10% Manual: 90%

Shift Left - Automation Facts

16

Tactical Solution

Requirements Design Development Testing Production

/Maintenance

Trad

itio

nal

Sh

ift

Left

Test Planning Automation Scripting / Execution

Test Planning

Scripting & execution Scripting & execution

API / Message Automating 30%

FIX Automating 20-30%

Behavior / Pattern Automating 20%

Combinatorial Automating 50%

GUI related Automating 30%

Regression Automating 40%

Better utilization of resources Early detection of issues Cost effective Fast time to market

32

Combinatorial Automation

17

Tactical Solution

Multiple test data combination Patterned behavior (Futures/Options/SWAP) Different Markets Month / Product Category / Commodity type Different session state (T or T+1)

Mapping Requirements Different workflow

Profiles Approval Schemas Algorithm validation

18

12

46

0

10

20

30

40

50

Volumes handled / Human Volumes handled / Robot

# o

f T

ran

sact

ion

s/d

ay

Volumes handled

2.4X Faster

1:3 Auto :

Resource

180% Productivity

Gain

40% Resource Reduction

35

12

0

10

20

30

40

Before Automation After Automation

Tim

e (

Min

s)

Average Handling Time

100%

40%

0%

20%

40%

60%

80%

100%

120%

Before RPA After RPA

# o

f F

TE

s

Resource Reduction

Before Automation After Automation

Tactical Solution

Shift Left - Automation Facts

Reactive to Proactive - Non Functional testing

19

Tactical Solution

Challenges • Workflow dependency - (multiple applications)

• Application speed • Application stability • Mimic Production scenario • Design for failure - (H/w, Process, Node)

Reactive to Proactive - Non Functional testing

20

Tactical Solution

Integrated Non Functional testing in Continuous Delivery

NF Testing Solutions

Detect H/w Failure & root cause analysis

Application profiling & root cause analysis

Response time

Throughput

Processor-utilization

Endurance

Interoperability

Node failure

Processor failure

Stability

Recovery

Achieving Resiliency - One Production Day Simulation

Member Test & Industry Wide Testing

21

Strategic Solution

Trading & Clearing APIs

Simulation with member/broking firm products

Replication of Critical path scenarios

Business Continuity scenarios

Performance / Failover & Fallback test

Dashboard & Reports to Regulatory body

Real time reference & Market Data

Test & Measurements

Infuse DevOps & Continuous Integration

22

Requirements/ User Stories

Code development Code review /

check in Build Package

Refresh (Data/Envi)

Functional Test

API test / FIX

Test Automation

Non Functional test

Defect Management

Reports / Dashboard

Deploy in Test Environment

Project Mgt, Collaboration, KM and

Reference docs

Tactical Solution

QEM

23

Strategic Solution

Effectively manage the quality (test) environment to achieve: Improved stability & availability Improved Quality Automation batch runs & easy maintenance Team/Member satisfaction

Automated provisioning of code deployment and easy maintenance of test environments in DevOps

Challenges • Tracking multiple changes in multi environments • Environment sharing with multiple teams for multiple verifications • Environment outage • Environment health check report

20-25% time loss

Blockers or surprises during testing

Quality Environment Management

Test env

QA env

Memb env

IWT env

Paradigm shift for test team

24

• Testers will need to understand the business goals & vision

• Testers become quality leaders within a multi-disciplined team

• Cross-skill and encourage flexibility

• Integrated test planning deliverables covering all teams & types of testing

• Utilize functional understanding to focus only on tests that deliver value

• Automated test execution in place from day 1

• Set targets, push for increased maturity & remove barriers

• Drive for continuous automation / integration

The role of testing becomes focused on technology and achieving business outcomes

Successful test teams will have to embrace continuous integration

Keys for success in a Mega-Testing Project

25

1. Commitment to Quality Culture (QA / QC)

2. Defect prevention and early detection

3. Understanding business and technical risks

4. Well defined ETVOX criteria – Entry Test Validation Output Exit

5. Continuous process improvement

6. Automate repeatable tasks

7. Standardize and simplify processes, tools, and methods

8. Manage by facts

9. End to End Customer Experience Testing

10. Build and maintain a high performing QA team

Paradigm Shift – Prevention over Detection

26

It needs a driver

Harmonize test planning to target testing to be effective & efficient

Collaboration amongst all teams & increased ownership by respective teams

Track in-flight quality & make value driven

Agility with continuous integration & Demonstrate reusability with test automation

Implement deep root cause analysis with a ‘one team’ mind-set

Skilled/champion staff

Pioneered New Processes and Tools

27

Thank you

Key Differentiators

29

Reduce test creation time by 30%

Improves functional coverage by 25% to 30%

Improves fault detection by up to 30%

Reduces maintenance work by 20%

Reduces end-to-end release cycle time by 50% or more

Standardizes test automation framework by stabilizing automation harness calls

Enables easier test framework migration

Helps reduce testing budget by up to 30%

Advantages

Modeling approach enables better system design archive and maintenance Shift Left approach, assures a considerable increase in software quality by capturing 100% requirements

coverage and reducing the cycle time The platform is ready and available to procure immediately – no perpetual software procurement delays

Requirements management • JIRA • IBM Door (RDNG) • HP ALM

Requirements Modeling

Test Execution

AUT

Repository Defect Tracking

Test Design

Test Harness

Code fixes Issues

Maintenance

Execution Reports

Test Coverage Reqs

Test execution • TOSCA • Selenium • OMX Batch runner

Modeling tool • IBM Door (RDNG)

Models Framework

Configuration

Code

Test Execution Test Design

Tactical Solution

18

Implementation Methodology

30

Manual Processes

Automation Workforce

Define Design Develop Test Deploy

Delivery Management Delivery Management Support & Change Mgmt.

Technical Infrastructure, Security and Governance

Ops SOP Build Feasibility

report FD/FS Config

Test Verification

Member Support

Security Policies.

Governance

Business Case

FD

FD/BRD UAT Ops Ready

Process Assessment

Tech Infra

Delivery Management

Program Governance, Security & Technical Infrastructure

= Business = Ops /Support = Test /

BA / SME

= Service Management

Our Improvement Approach

31

Test Governance PEOPLE PROCESSES TOOLS

Sanitize Stabilize Optimize

Driving a need to transform testing

• Transforming skills set • Building relationship • Making it competitive • Testing culture shift • Strong automation

team • Increase awareness

• Process Assessments • Early testing • Everyone’s

responsibility • Broader test coverage • Frequent/apt

communication

• New infrastructure • Making it accessible • Usage of tools • Reusable codes • Dashboard reporting • One click automation

(code deployment)