Taming the Beast: Test/QA on Large-scale Projects

W7 Test Management

5/1/2013 1:45:00 PM

Taming the Beast: Test/QA on Large-

scale Projects

Presented by:

Shaun Bradshaw

Zenergy Technologies, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Shaun Bradshaw

Cofounder of Zenergy Technologies, Inc., a QA and agile solutions consulting firm (zenergytechnologies.com), Shaun Bradshaw has spent the past fifteen years advising, teaching, and mentoring clients to improve their QA and test processes with effective testing and test management techniques. He is the coauthor, editor, and trainer of a suite of methodologies covering testing, test management, and test metrics. A popular speaker at many of the major QA industry conferences, Shaun is known for his presentations on test metrics, the S-Curve, and the Zero Bug Bounce.

4/16/2013

1

AGENDAAGENDAAGENDAAGENDA

• What constitutes a large-scale project?

• QA Approach

• Data Strategy

• Environments / Infrastructure

• Team Structure

• Communications

• Integration Testing

• Performance Testing

• Key Take Aways

4/16/2013

2

LARGELARGELARGELARGE––––SCALESCALESCALESCALE PROJECTSPROJECTSPROJECTSPROJECTS

• What constitutes a “largeWhat constitutes a “largeWhat constitutes a “largeWhat constitutes a “large----scale project”?scale project”?scale project”?scale project”?

� Number of resources?

� Duration?

� Number of integrated components?

� Cost?

� Other?

LARGELARGELARGELARGE----SCALESCALESCALESCALE PROJECTSPROJECTSPROJECTSPROJECTS

• What constitutes a “largeWhat constitutes a “largeWhat constitutes a “largeWhat constitutes a “large----scale project”?scale project”?scale project”?scale project”?

� Number of resources? 100+

� Duration? 2+ years

� Number of integrated components? 6 (major)

� Cost? $70 million

� Other? Multiple external vendors operating in

strict waterfall methodology

4/16/2013

3

TERMINOLOGYTERMINOLOGYTERMINOLOGYTERMINOLOGY

• System TestingSystem TestingSystem TestingSystem Testing

� Performed at the component level

� Includes functional, data transformation/validation, and security tests

• Integration TestingIntegration TestingIntegration TestingIntegration Testing

� Performed across multiple components of the solution

� Includes integration, data quality validation, and cross-component security tests

QAQAQAQA APPROACHAPPROACHAPPROACHAPPROACH

• Align with overall development methodologyAlign with overall development methodologyAlign with overall development methodologyAlign with overall development methodology

� Don’t create “culture shock” by following completely different test methodology� If the project is waterfall, utilize the V-model of QA or

something similar

� If the project is agile, help establish iterations that QA can work with and ensure open collaboration across the project

� Ensure the QA approach is thoroughly communicated across the project team

4/16/2013

4

QAQAQAQA APPROACHAPPROACHAPPROACHAPPROACH---- PHASEDPHASEDPHASEDPHASED TESTINGTESTINGTESTINGTESTING APPROACHAPPROACHAPPROACHAPPROACH

• Here was our approachHere was our approachHere was our approachHere was our approach

QAQAQAQA APPROACHAPPROACHAPPROACHAPPROACH---- HIGHHIGHHIGHHIGH LEVELLEVELLEVELLEVEL TIMELINETIMELINETIMELINETIMELINE

4/16/2013

5

DETERMINE THE DATA STRATEGYDETERMINE THE DATA STRATEGYDETERMINE THE DATA STRATEGYDETERMINE THE DATA STRATEGY

• Data AcquisitionData AcquisitionData AcquisitionData Acquisition

� Create test data � From component systems

� Based on test criteria

� Pull data from production� Ensure proper data quality

� Data meets the needs of the testing

� Data is in “proper” form

� Data masking?

� Data Volume?

ENVIRONMENTS/INFRASTRUCTUREENVIRONMENTS/INFRASTRUCTUREENVIRONMENTS/INFRASTRUCTUREENVIRONMENTS/INFRASTRUCTURE

• Determine, as early as possible, what environments Determine, as early as possible, what environments Determine, as early as possible, what environments Determine, as early as possible, what environments

are available & necessaryare available & necessaryare available & necessaryare available & necessary

� Component-level system test environment

� Integration test environment

� Performance test environment (possibly

production if this is the first instance of the

application)

� UAT environment

4/16/2013

6

TEAM STRUCTURETEAM STRUCTURETEAM STRUCTURETEAM STRUCTURE

Key RolesKey RolesKey RolesKey Roles

� Test Architect – Creates overall test strategy

communicates/socializes it to project team

� Test Manager – Manages day-to-day test activities

� Test Leads – Component-based/Technology-based

� Test Analysts – Develop and execute tests

� Test Project Manager – Ensures test activities are

properly scheduled

� Integration Lead – Pulls together integration test effort

TEAM STRUCTURETEAM STRUCTURETEAM STRUCTURETEAM STRUCTURE

• Key Roles Key Roles Key Roles Key Roles (continued)(continued)(continued)(continued)� Performance Lead/Engineers – Develops performance

test plan, scenarios, and executes� Test Data Modeler – Determines data needs &

acquires test data� Business Leads– provide user-type input/review of test

scenarios and test cases� Technical Leads – Assist in creation of QA environment

and data needs� Release Manager – Helps manage all configuration

and component movement

4/16/2013

7

COMMUNICATIONSCOMMUNICATIONSCOMMUNICATIONSCOMMUNICATIONS

• The BIGGEST Challenge for QAThe BIGGEST Challenge for QAThe BIGGEST Challenge for QAThe BIGGEST Challenge for QA

� Utilize 360° communication strategy� QA approach and reasoning to upper management (early

and often)

� Resource and schedule requirements to peers

� Tactics, techniques, and goals to down line resources

� Beware of assumptions made due to lack of

communication

COMMUNICATIONSCOMMUNICATIONSCOMMUNICATIONSCOMMUNICATIONS

• The BIGGEST Challenge for a LargeThe BIGGEST Challenge for a LargeThe BIGGEST Challenge for a LargeThe BIGGEST Challenge for a Large----Scale ProjectScale ProjectScale ProjectScale Project

� Identify the right resources to include when

discussing� QA Approach / Test Strategy

� Data Requirements / Strategy

� Resources & Schedules

� Test case framework

� Test phases

4/16/2013

8

INTEGRATIONINTEGRATIONINTEGRATIONINTEGRATION TESTINGTESTINGTESTINGTESTING

• The 2The 2The 2The 2ndndndnd BIGGEST Challenge for a LargeBIGGEST Challenge for a LargeBIGGEST Challenge for a LargeBIGGEST Challenge for a Large----Scale ProjectScale ProjectScale ProjectScale Project

� Logistical challenges

� Political challenges

• Purpose Purpose Purpose Purpose –––– demonstrates that the IT processes and demonstrates that the IT processes and demonstrates that the IT processes and demonstrates that the IT processes and

systems/components built to support business systems/components built to support business systems/components built to support business systems/components built to support business

processes are mutually integrated correctlyprocesses are mutually integrated correctlyprocesses are mutually integrated correctlyprocesses are mutually integrated correctly


• Integration Test ApproachIntegration Test ApproachIntegration Test ApproachIntegration Test Approach

� Map out critical data flows between components (IDEF is a good model)

� Determine “day in the life” scenarios ensuring that all critical data flows are covered

� Develop integration test cases using previously created system test cases to minimize rework (use a modular test framework)

� Utilize a “bottom-up” approach where possible (start with C2C scenarios, then E2E scenarios)

4/16/2013

9

INTEGRATION TESTINGINTEGRATION TESTINGINTEGRATION TESTINGINTEGRATION TESTING

DRAFTDI

(Includes

COPS/RDM)

DW

(ODS/ODM)

BI

NXG

Ess

Base

PS

Formatted

Data to

JGen

(Batch

Data)

Combo Edit

Explosions

SIF Formatted Files

Direct to E2

Formatted Files

Processing Schedule

Attachments

Journal Line Attachments

Hyperion Out:

Actuals and

Budget

Hyperion In:

Budget Data

GL91I0014

Webservice PS Chart of

Accounts (partial or

complete) to be validated

by DI (PCR 33)

Reports

PeopleSoft UI

Spreadsheet Uploads

Converted e2

to PS Data

Manual Data Entries

Data Dictionary / SIF

NXG

Balance

Life Com

COPS Audit Info

Application and

User Security

Internal ControlsBusiness Rules

Rejected File

Self Service Balance

(Daily Balance File)

Config/Settings:

Open Periods, Mapping

Rules, Ledger Code,

Chart Fields, Open

Periods, Trees, Business

Units

JE Lines/

Hdr

(Nightly

Batch)

AutoSys

Excel Active Directory

PS Query

Results

E2 Mapping Rules

Black Box e2

Formatted Files

E&Y

CIMS

Server/Mainframe

Environment

Business Rules

Error Theshold

E2 to PS Chart field

translation - flat file

Input (PCR 15)

GLCDS

Converted Rejection Info

Email

Fin Detail

Table (Nightly

Batch)

Compass Integration Model A1101711 1545.vsd

Date Dimension File (On

Demand)

Via Data Analyst Package

to RDM

Budget

On

Demand

(ETL)

Financial

Transactions

File (FTF)

E2 Balances

Manual Upload

Of Opening /

Outstanding Items

Match/Pass RulesRemitter

Lookup

Table

Account

Structure

Extract

Extract

Criteria

Template

Manual

Upload

Outstanding

Template

Trigger

for FTF

Trigger for

Balance

Reconciliation and Reporting Data

Common View CSA

(Webpage)

PS Universe

(Read Only)

DI Staging Tables

(Read Only)

ODS/ODM/RDM

Universe (Read

Only)

COPS Universe

(Read Only)

Non Chart Field Table

(Excel File) Via Data

Analyst Package to RDM

Stop

File

VPD

Security

Oracle

Webservice NDM

Informatica

Self

Service

Business

Object

Reports

Self

Service

EssBase

Reports

80 Compass

Developed

Reports

BO XIR3 Oracle DBAs

BO Portal

(Internet)

Financial ControllerHyperion EssBase

9.3.3 (Arc Cube) Consumers

InfoView

Oracle

Excel Add-in

Accurate Webclient

Informatica

Business Objects

Commonview/CSA

Prod Application

Admin

Informatica

DA Tool Email Server

WLK Day

Transactions

Outstanding Items

Conversion Data

Remitter

Number Table

Excel

(min

2007)

Perl and

Korn Shell,

Acs Scripts

NXG BO

Universe

Reports

NXG Admin

NXG Reconciler

BO User

Web

Driver

Generator

Ariba

JE Lines/Hdr

Ledger

(Trickle

Feed)

PeopleSoft Autosys

Adapter

SFTP

JGen Mappings

Users

RDM

Universe

(Read Only)

Webservice

Validation

Data (PCR

33)

ETL Controls

for ODS data

movement: All

but ODM data

EIS Trees (ARC)

EAS Ledger

Table Data

(ARC)

Date

Dimension

CDS

GL

Data Analyst

Admins

Admins

PS Users

Manual Open/Close Process

Processing Schedule

Processing Schedule

Excel Upload Template

Date/Time

Manual Schedule

Processing Schedule

Date/Time

Internal Controls

Internal Controls

Internal Controls

Internal ControlsProcessing Schedule

Date/Time

Application and

User SecurityApplication and

User Security

Application and

User Security

Active Directory

System

Connectivity

Oracle InformaticaOracle Active Directory

AutoSys

System

Connectivity

RDM

01

02

03

040506

01

02

03

04

05

30 31 1011

13

1514

12

1514

01

02

03

010203

040506

10

11

30 31 32 10

30 31 32

1617

10

11

12

30 31

03

01

02

04

10111213

1415

01

03

Application and

User Security

Server/Mainframe

EnvironmentSystem

Connectivity

Server/Mainframe

EnvironmentSystem

Connectivity

AutoSys

InformaticaOracle Active Directory

Server/Mainframe

EnvironmentSystem

Connectivity

AutoSys

Webservice NDM

Server/Mainframe

EnvironmentSystem

Connectivity

AutoSysServer/Mainframe

Environment

System

Connectivity

Active Directory

Consumers

Shared File

System

Validation Error Report Shared File

System

Translated PS File

Admins

Cost Center Validation

Cost Center BU Account

Detail Extract via SFTP

PS

Upstream

Sources

Upstream

Sources

Upstream

Sources

e2

e2

NXG

Team

e2

Annuity

Team

BI Team

Journal Entry

Preparers

Accurate

WebClient

NXG Data Base View

Author: Alan Smith

Trigger

for FTF

Self Service Balance

(Daily Balance File)

Tree Files

Balance

Files

BO Reports

SQL Data

Load

CDS

ICAS

Cost Center Validation

Data

Modified ICAS Report

Excel File Manual Load

Policy Level Claims

Reports

Processing

Schedule

Hourly JE extract for spreadsheet upload

Policy and Company Number

Business Objects

Treasury

Tax

Invest

Accntg

Trigger for

Balance

01

0203

30 31

01

30 31

30 3101020304

Server/Mainframe

Environment

System

Connectivity

Webservice

Controls

for Trickle

Feed and

Batch for

JE Lines

etc. data

movement

Journal Entry

Preparers

Journal Entry

Preparers

12

04

32

Send controlled by DI but

does not go through DI

13

BI Team

Notes

1617

40

111213

1840

33

40

30

40

32

40

33

02

05

10

40

10

40

32

40

32

40

1718

How to read the model

ACTIVITY

A

ACTIVITY

B

A Sends to B

B Pulls from A

SourceInput

DestinationOutputPrimary Flow

Controls, Contraints,

Rules, Timings

Mechanisms,

Systems, Users

Mechanisms and non-data generated

Controls grouped to remove clutter of lines

Red lines/text indicate High Risk

Green line is for Reports only

Dotted line is for PCR 33 only

Labels,

Descriptions

10ICOM

Number

Input 01-09

Output 10-19

Control 30-39

Mech 40-49


• Validation ApproachValidation ApproachValidation ApproachValidation Approach

� Focus on positive scenarios during integration

(use system testing for most of the negative test

scenarios)

� Determine the best approach given time,

resources, and risk� Validation of Execution

� Validation of Results

4/16/2013

10


• Validation of ExecutionValidation of ExecutionValidation of ExecutionValidation of Execution� Assumes detailed validation of expected results was

completed during system testing and system test scripts are being re-used as part of integration

� Generally, the most efficient method of validating results for integration scripts, but increases the risk of missing potential defects

� Best used when a component has been thoroughly system tested - it is acceptable to simply verify the process executed


• Validation of ResultsValidation of ResultsValidation of ResultsValidation of Results

� Doesn’t assume the correctness of results from

previous testing phases

� Necessary to fully validate the process outputs at

a detailed level via external calculations (i.e.

“tool”, spreadsheet, SQL query, manual

calculations, etc.)

� Although dependable, low risk this method

time/resource intensive

4/16/2013

11


• ExecutionExecutionExecutionExecution

� Its all about COMMUNICATION

� Because an individual tester may not have

requisite knowledge to validate every step of an

integration scenario “hand-offs” are necessary� Assign a resource to manage the communication of the

hand offs (we don’t want tests sitting)

� Hold daily triage and schedule meetings

� Be flexible ☺

PERFORMANCEPERFORMANCEPERFORMANCEPERFORMANCE TESTINGTESTINGTESTINGTESTING

• Application PerformanceApplication PerformanceApplication PerformanceApplication Performance

� Verify each component is properly tuned as early

as feasible

� Align with system test phase

• Infrastructure PerformanceInfrastructure PerformanceInfrastructure PerformanceInfrastructure Performance

� Utilize an integrated environment to determine

issues in the infrastructure of the solution

� Align with integration test phase

4/16/2013

12

KEY TAKE AWAYSKEY TAKE AWAYSKEY TAKE AWAYSKEY TAKE AWAYS

• Develop an overall test strategy that aligns with the development methodology and culture of the organization

• Ensure key roles are properly filled� QA Architect� QA PM� Business Owners� Data Modeler

• Determine where/how you will get your data (the data strategy)

KEY TAKE AWAYSKEY TAKE AWAYSKEY TAKE AWAYSKEY TAKE AWAYS

• Don’t rush to integration testing

• Identify key business and technical resources to assist in developing the integration scenarios

• Utilize a modular testing framework so creation of integration tests requires little additional work

• Performance test key components as early as possible and ensure the entire solution is performance tested

• Communicate, communicate, communicate

4/16/2013

13

Questions?Questions?Questions?Questions?

Thank you!Thank you!Thank you!Thank you!