InfoStretch - Retention of Test Evidence

All trademarks are the property of their respective owners.©2004-2010 InfoStretch Corporation. All rights reserved.

Retention of Test EvidenceGuidelines, Approaches, Recommendations Financial Industry

February 2011

Presenter: Sivakumar AnnaDirector of Enterprises QAInfoStretch Corporation

1. InfoStretch Overview2. Financial Industry Requirements3. Guidelines 4. Approaches5. Recommendations

Agenda

INFOSTRETCH CORPORATE(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected]©2004-2010 InfoStretch Corporation. All rights reserved.

phone: (408) 727-1100 | website: www.infostretch.com | email: [email protected]

©2004-2011 InfoStretch Corporation. All rights reserved.

InfoStretchOverviewElevating Confidence

1

Organization HQ in Silicon Valley Offices in New York, Seattle 105+ Clients and 160+ Projects Well Established Recruitment, retention and training

programs

People Average 6 years experience 250+ Engineers/Consultants Industry Expertise in Finance,

Retail, Mobile and Healthcare Strong Project Mgmt. team

Infrastructure Extensive Platform

Compatibility and Performance Test Labs

2000+ Mobile Handset Test Lab ISO 9000-2001 Certified High Availability and 24 hours

secure offshore centers with ISO 27001 Audited

Overview

INFOSTRETCH CORPORATE(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected] | Presenter: Rutesh Shah©2004-2011 InfoStretch Corporation. All rights reserved.

Partners

mailto:[email protected]

RecognitionUnique Perspective – Leadership Team with Entrepreneurial, Startup, Fortune 500, Top Technology & Big 4 Consulting Firms Experience.

Long Term Relationships – Focus on building long term relationships and competencies around client 100% reference-able. Clients including T-Mobile, AT&T, Amazon, Intel, TNS, Harmony & Wells Fargo

Strong Growth – Strong Year over Year Growth and Profitable Operation Since 2004. Significant Investments in Platform Compatibility and Device Labs

(p) (408) 727-1100 | (w) ww.infostretch.com | (e) [email protected]

19th Fastest growing privately owned

companies in Bay Area

Awarded Inc

500/5000 company

of the year 2010

Among10 Most Promising

Testing Services Company

Gartner Top 20 QA Service

ProviderExceptional QA

Services and Solutions

GESIA Rising Star Award 2010 – Best

Software Product

157th Fastest growing Technology company in USA

http://www.inc.com/inc5000/2010/index.html

http://www.thesmarttechie.com/index.php

http://www.gesia.org/index.php

Retaining Test EvidenceFinancial Industry

2



Background Retention of Test Evidence

What

Why

How

QA Test Execution Logs Environment Information Team Information Process Followed

SOX Compliance Service Level Agreement Risk Management Out-Of-Bound / Multi level Authentication

Guidelines Approaches Tools & Technologies Recommendations

GuidelinesFor Approach Selection

3

Guidelines:

Evidence Collection ProcessTest Case Selection ProcessEnvironment Selection

Information to be Retained

Storing & Retrieving of Evidence



Test case Selection - Guidelines

Evidence Collection Easier to adopt Minimal Training

Test Case Selection Critically factor Risk Based Selection Coverage Based

Environment Selection

QA Environment User Acceptance

Testing (UAT Pre-Prod Testing

Information to be Retained

Document Archives Test Logs Stake Holder Sign-offs

Storing/Retrieving Evidence

Archival Periods Locations/Accessibility Access Control



Teams Interactions

DEV* Code instrumentation* System Logging* Server Logs

QA* Execution Records* Test Coverage* External System - Mock

UAT* Compliance test cases* End-to-End Scenarios* External System

Audit / Risk* Periodic review of compliance* Enhance test scope

Prod* Review production issues* Feedback to QA/Audit team

ApproachesAnalyzing the different

methods in Test Retention…

4

Approaches:

Manual Testing

Semi-Automated TestingAutomated Testing

Backend Snapshots

Application Instrumentation

Record test evidence in terms of screenshots or movies using external

tools during manual testing.

Manual Testing:

PRO’s Capture Information as needed

Timed Snapshot Captures /

Videos

Best for new features

Cost effective

ExamplesApproaches – Manual Testing

CON’s Capture Information as

Needed Timed Snapshot

Captures / Videos Best for new features Cost effective

Time consuming

Risks due to manual errors /

missing the captures

Not suited for regression

Training and Implementation

Challenges

Additional steps

Increases test execution time

ExamplesApproaches – Manual Testing

Manual testing with automated tool to collect test data evidences.

Semi-Automated Testing:

PRO’s No change in process

Evidence collection is

automatic

Efficient test runs for evidence

data collection

Additional benefits of QC 11

Features (Mirroring, Data

Handling, …)

ExamplesApproaches – Semi-Automated Testing

VSTS Test Runner

HP-Sprinter (part of HP QC 11)




Training and additional

licensing cost

Quality Center upgrade is

needed

Tools Inter-operability

ExamplesApproaches – Semi-Automated Testing

VSTS Test Runner

HP-Sprinter (part of HP QC 11)

This approach gives user the facility to collect test data evidence using tools

for automation.

Automated Testing:

PRO’s Reduces Manual effort

Saves time, faster QA Cycles

Simultaneous testing across

multiple platforms

Best suited for Regression

testing

Helps to cover large test

matrix

Framework based approach

Faster ROI ExamplesApproaches –Automated Testing

HP QC-QTP




Skilled resource needed to

automate test cases

Not suited for new features

Maintenance of test

framework and Test cases

Limited End-To-End test

coverage involving backend

legacy systems

ExamplesApproaches –Automated Testing

HP QC-QTP

Snapshot (DB, Server, Logs, Files, etc) taken during test approach for

evidence.

Backend Snapshot:

PRO’s Controlled data environment No overhead on tester Easier to implement Captures all transactions and the related data

Approaches – Backend Snapshot

CON’s Maintenance cost Selective test coverage Retrieving the necessary information is difficult Co-ordinate with other team members Need to write tools to parse and analyze details

Instrumenting the application to properly monitor application level

transactions

Application Instrumentation:

PRO’s Control over data collection for evidence. Obtain data not available through external tool Capture Transactional level details (HTTP,

Application Server Level, DB Level)

Approaches – Application Instrumentation

CON’s Time consuming Skilled resource Complicated process Development Team’s Responsibility May have an impact on application performance



Testing Approach Matrix

Application Instrumentation

High level of Detail

Lower level of Detail

High Costs

Lower Costs

Backend Snapshots

Automated Testing

Semi-ManualTesting

Manual Testing

Increased test evidence requirement levels

require more powerful tools.

RecommendationsChoosing the best method.

5



Recommendations – Short, Mid, Long Term

Manual Testing•Short Term•1 to 3 Months

• Easier to implement• Satisfy the current audit process• Review the recordings and confirm

with internal auditing team• Fine-tune the process as the time

progress• No major investment

Semi Automated Testing• Mid Term• 4 to 6 months

•Benefits of upgrade to QC 11

•Newer process with additional tool features

•Integrated in to the manual testing

•No specific training / steps required from the testing team

Automated Testing• Long Term• 6 to 9

months

• Standard benefits of Automated regression

• Evidence collected for more set of test cases

• Centralized approach for evidence collection



Case StudyOur Client

The Challenge

Leader among Financial Institutions Distributed QA Team Test Evidence collected by Business / UAT Team Lack of Tools and Technologies

Our Approach

Frequent Releases (Every 3 weeks) Large Test Artifacts (20,000+) Risk Management Team Requirements Audit Review Team Requirements

Identified a subset of test cases for evidence and compliance (~1150) Short Term – Semi-Automated Test Evidence Collection (Deployed HP QC 11 with HP

Sprinter) Long Term – Test Automation with Application Instrumentation Artifacts Collected (Screenshots, Logs, Environment, Test Scripts and Test Data, etc)

60% - Automation in QA25% - Using Manual / Sprinter15% - UAT by Business Team

Santa ClaraHeadquarters2880 Lakeside Drive, Ste 200Santa Clara, CA 95054(408) 727-1100

Partners

New York17 Tanglewood Rd,Scarsdale, NY 10583(703) 395-9791

Ahmedabad5th Floor, Regency Plaza, Anand Nagar, Ahmedabad-15, Gujarat, India91-79-40266266

MumbaiVashi InfoTech Park, B103, Tower # 1, Vashi Rail Station, New Mumbai, Maharashtra, India91-22-6947900

Website: www.infostretch.com



34

Test Management with InfoStretchHosted by PushToTestPresenter:Manish Mathuria, CTO InfostretchFrank Cohen, CEO PushToTest

All trademarks are the property of their respective owners.©2004-2011 InfoStretch Corporation. All rights reserved.

InfoStretch Webinar Series Presents:

Mark your Calendar17th Feb 2011

17th Feb 2011

Go to InfoStretch.com for more information

Go to InfoStretch.com for more info.