Alex syvorotka - QA: Customer Oriented Testing Approaches in theory and practice

CUSTOMER ORIENTED TESTING APPROACHES IN THEORY AND PRACTICE

by Alex Syvorotkaformer SQA @ Lohika

2011

Lohika Confidential

Contents

• Presentation Description and Short words intro

• HP UCMDB Quick Overview

• COST aims

• Methodology

• Program stages

• On-Site visit

• Program requirements

• COST testing

• Met goals

08/20/11 Lohika Confidential 2

Presentation Description

• There are many approaches in Quality Assurance are being used in enterprise software development which include many different testing types. Starting from classic functional testing and going beyond with different techniques of non functional testing.

• This presentation is intended to share the experience around customer oriented testing, aka COST, gained in one of Lohika projects – HP UCMDB.

• It will describe how important such approach is, what are the outcomes and which tools, processes and techniques can be used.


Short words intro

• Every workday QA tries to provide testing coverage of software components by test cases as full as it possible. In order to make coverage really high HP has agreement with premier customers to make EA cycles during on-site visits. And along with other benefits of such visits - to retrieve dumps of their software in production. Obviously owing to security reasons customer hides vulnerable details such as credentials, business critical pieces of information, etc.

• QA analyses DB dumps to clarify in what way (HOW?) particularly a certain customer uses HP UCMDB (BSM). QA creates tables\maps of: DB capacity, used configurations, amounts. This allows to use replicated customer’s environments for testing and also to understand which of the configurations is the most effective for a specific area of testing coverage.

• Cross-team testing – a method applicable to mass testing of software product components. The main idea is to make a joint effort of all teams to cover their areas of responsibility. An experienced QA guy is being challenged as a cycle owner. He prepares cycle time periods, focus points, servers, presentation. And, if necessary, sheds a light onto possible technical issues and “bottlenecks”. The owner assigns details of the cycle and tasks to participants and sets timelines. As a result of cross-cycle an aggregated report is generated and sent to management of the particular area.


HP UCMDB Quick overview

• The complexity of today’s IT infrastructure places enterprises at great risk of prolonged and frequent IT outages as a result of change. For example, market research has found that upwards of 80 percent of IT outages are caused by human error that stems from both planned and unplanned IT changes, as well as shortfalls in testing and process inadequacies.

• Yet simply avoiding change is not possible. Things break. Software needs patches. Servers will always need memory upgrades.

• Agility in IT means responding to business needs when a company builds, buys, divests or merges lines of business. It is defined by the ability to quickly deploy and manage new services for example, a Web application that the business launches in order to pursue an emerging market opportunity and stay ahead of the competition. IT’s role as a strategic business partner stems from its ability to make changes quickly – and less risk.

• If change is mandatory, then IT organizations need a means to control change and manage the impact of change when it does occur. This is the value proposition of a CMDB.


Painful process called “Change”


• A Configuration Management DB visually models information about IT infrastructure components in order to understand the interdependencies among these components in the context of IT service.

• If IT understands the relationships among its IT components – if it has the ability to map components to applications to the business – then IT can control change and manage impact. IT needs to understand the impact of change prior to implementing changes in their production environment.

• For example, IT needs to understand impact the installation of a new O/S software patch to a single box in a server farm will have on a business service – such as online trading – and can take proactive steps to mitigate the risk of an outage. Similarly, should outages arise from unplanned changes or for any other reason, such as fault failures, the relationships the CMDB maps allows IT operations to quickly find the root cause of IT outages or degradations – often before performance, availability or end-user satisfaction are adversely affected.


CMDB’s Purpose



CMDB and discovery relationships







So…


COST aims


• Test products from the customer’s perspective, using real business scenarios executed on authentic data

• Provide version status as reflected on customers

• Bring customer’s use cases expertise in house to improve prior release testing

COST Methodology

• Understand customers usage, needs and pain

• Learn customers methodology and implementation

• Simulate customers deployments using real data

• Execute customer oriented scenarios

• Focus R&D on the real usage of our products


COST Program Stages


• Visit customer site

• Study Product implementation

• Understand the organization testing process

• Receive feedback/enhancement requests directly from product users

• Retrieve Product Dump

Learning

Phase

Environment Simulation Upgrade E2E cycle Beta Program

COST Program Stages Cont.1


• Replicate customer environment using real data

• Document business processes and usage scenarios

• Validate information

Learning Phase

Environment

SimulationUpgrade E2E cycle Beta Program



Learning Phase

Environment

SimulationUpgrade E2E cycle Beta

Program

• Upgrade environment and data

• Run user scenarios

• Validate new features



Learning Phase

Environment

SimulationUpgrade E2E

cycleBeta

Program

• Install new version in QA Laboratory

• Perform upgrade over customer’s dumps

• E2E cross teams testing

• Massive bug hunt on customer’s dumps



Learning Phase

Environment

SimulationUpgrade E2E cycle Beta

Program

• Install new version on site

• Perform upgrade

• Post upgrade tests with users

On Site Testing Activities

• Selecting areas for ‘on site testing’

• Critical areas that a large percent of customers are exposed to.

• Areas containing high number of fixes or where refactoring has been done (risky areas).

• Areas that suffer from ‘bad reputation’ (are more prone to defects)

• By demand – from QA or Dev



Cont.1• Planning the testing activity

• Analyze the feature with relevant QA FL and Dev FL; understand current problems and risk factor related to them.

• Get buy in of Dev (CORD, feature team) and set criteria for build that will be used for this activity (including fixing QA holders)

• Determine environment coverage needed for this feature (DB, OS and etc.)

• Identify actions that can be done on customer site and verify that these actions cover the risk areas.

• Create specific use cases to be tested at each customer

• Learn the feature thoroughly including troubleshooting knowledge

• Select between remote activity and on site activity

• Set available support level for customers participating in this activity.02/20/09 Lohika Confidential 20


Cont.2• Selecting customers – requirements and

profile

• Customer characteristics: environment parameters (if relevant to tested feature), projects structure, special configuration, challenges and problems and etc.

• Customer is willing to cooperate and accepts relevant requirements

• Match between feature and usage patterns at customer site.

• Location (when relevant)



Cont.3• Execution phase

• Before conducting the activity make sure all preparations are done at customer site.

• Log all defects and problems detected on site, make sure to take all relevant data (logs, configuration).

• If possible, obtain a copy of production project.



Cont.4• Wrap up

• Maintain support level as agreed with the customer before the activity

• Wrap up communication with customer – provide status of issues detected on his environment.

• In case of open Support Requests – involve CSO and CORD.


COST Testing Process

• Requirements definition (obtained during customer’s visit)

• Installation and configuration

• Upgrade

• Critical business scenarios per project

• Testing scenarios and features validation

• Business flows by user role

• E2E testing


Met goals

• Get overall understanding of HP UCMDB implementation at the customer.

• Understand challenges faced by the customer.

• Define specific testing requirements for upgrade and critical areas in HP UCMDB.

• Obtain project specific business processes to be translated into testing scenarios.

• Create somewhat like Customers Knowledge Base


Q&A?

WELCOME TO LOHIKA!

Lohika Confidential 2608/20/11

Technology

Alex syvorotka - QA: Customer Oriented Testing Approaches in theory and practice