Managing Independent Software Managing Independent Software Testing Project Testing Project -- A Case Study A Case Study from Multimedia Super Corridorfrom Multimedia Super Corridor
Khor Cheng San (SADC Manager)Khor Cheng San (SADC Manager)Azdilizan Abdullah (Senior Executive)Azdilizan Abdullah (Senior Executive)Solution Assessment and Development Centre (SADC),Solution Assessment and Development Centre (SADC),MSC Innovation CentreMSC Innovation Centre2 August, 20052 August, 2005
MSC Innovation Centre
Copyright Protected 2005 MDC
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Table of ContentsObjectives of This Case StudyIntroduction to Solution Assessment and Development Centre (SADC)Test Project Management (TPM) MethodologyTest Project Management (TPM) ProcessesQuality Management System (QMS) FrameworkTest Methodology Framework (TMF)Case Study: Test Project OverviewCase Study: Test Project PhasesCase Study: Lessons Learned
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Objectives of This Case Study
Present our experience in using PMBOK® and CMMI combination with our own testing processes that is the SADC Test Methodology Framework (SADC TMF) in managing independent software testing services to the stakeholders.
The SADC TMF encapsulates the various components required for test execution and test management, and is the basis for conducting Test Projects in the SADC.
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
SADC offers independent application verification & validation services, at a fraction (“cost-effective”) of the cost of building and maintaining an internal software test/QA department. Our Added-value proposition are:
Objective analysisFills the gap in staff resource Assures the delivery of reliable & robust application with proven SADC test methodologies and processes (QA/QC functions) “Do what they do best”
Solution Assessment & Development Centre (SADC)
“The solution for product perfection”
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Solution Assessment & Development Centre (SADC)
“The solution for product perfection”
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Solution Assessment & Development Centre (SADC)
“The solution for product perfection”
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Test Project Management (TPM) Methodology
The SADC’s Project Management methodology applies recognized project management methods and techniques that are built upon
Project Management Institute's Project Management Body of Knowledge (PMBOK®) concepts. Software Engineering Institute’s Capability Maturity Model®Integration (CMMI) model.
This PMBOK® and CMMI combination culminates in the SADC’s Quality Management System (QMS) Framework and Test Methodology Framework (TMF) is to provide the basis for the SADC’s Test project teams to carry out their work in a well-structured manner to produce high quality output.
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Test Project Management (TPM) Processes
The concept of our Project Management methodology is illustrated below:
INITIATIONPLANNING
CONTROL
EXECUTION
CLOSURE
Test Project Management Processes
The Test Project Management Processes of Initiation, Planning, Execution, Control and Closure are the type of processes occurring throughout a test project, keeping in balance the competing demands for scope, time, cost and quality.
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Test Project Initiation
Initiation is the process of formally authorizing a new test project or that an existing test project should continue into its next phase.
Test Project Management (TPM) Processes
Test Project Planning
Planning is an ongoing effort throughout the life of the test project and is considered the most important phase in test project management.
Test Project Execution
Test Project execution will be guided by the test plan developed in the Test Research/Analysis/Design Phase.
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Test Project Management (TPM) Processes
Test Project Control
Test Project control will involve the regular review of metrics and report status in order to identify variances from the test planned project baseline so that corrective action can be taken when necessary.
Test Project Closure
Test Project closure is the last major phase of a test project’s life cycle. It is performed once all stated test project objectives have been met and the client has formally accepted the test project’s deliverables.
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Quality Management System (QMS) Framework
The QMS contains a set of processes, guidelines, templates, checklists and forms that will be tailored for the purpose of the Test Project.
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Test Methodology Framework (TMF)
SADC Test Methodology Framework (SADC TMF)
SADC SADC TMFTMF
TestProcesses
Procedures & Templates
Test Methodologies
Standards& Guidelines
CM DatabaseOf Test Projects
SADC-CMDefect Management
SADC Test Methodology Framework (SADC TMF)
SADC SADC TMFTMF
TestProcessesTestProcesses
Procedures & TemplatesProcedures & Templates
Test MethodologiesTest Methodologies
Standards& GuidelinesStandards& Guidelines
CM DatabaseOf Test ProjectsCM DatabaseOf Test Projects
SADC-CMSADC-CMDefect Management
Defect Management
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Documentation Components of SADC TMF
SADC Test Management Framework: Documentation Artifacts
SADC Test SADC Test MethodologiesMethodologies
SADCTest Methodologies
SADCConfigurationManagement
SADC Standards andGuidelines for
Test and Integration
Template Documents forSADC Test Projects
CM Database ofTest Projects
SADC Test Management Framework: Documentation Artifacts
SADC Test SADC Test MethodologiesMethodologies
SADC Test SADC Test MethodologiesMethodologies
SADCTest Methodologies
SADCConfigurationManagement
SADC Standards andGuidelines for
Test and Integration
Template Documents forSADC Test Projects
CM Database ofTest Projects
SADC Test Methodologies contains details of the Test Methodologies to be employed in the test execution and test management of TestProjects in the SADC.
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Industry / Market – Airline
The Challenge – Client A wanted to performance test their Web Based System with realistic workloads to determine whether the system can handle the projected user load of 600 concurrent connection (10% of total user load) with a total estimated user base of 7200 crew (1200 Technical Crew and 6000 Cabin Crew) .
The Solution - SADC uses Rational Suite TestStudio™ to performance test Client A Web Based System with incremental concurrent user load of 1, 50, 100, 250, 500, 600 and 750 (to stress the system) virtual users. The performance test results will enable the client to accurately predict how their critical (widely used) functions will perform under projected user load.
Case Study: Test Project Overview
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Testing HighlightsPerformance testing ensures proper scalability and identifies other performance problems in a real-life business environment.Performance testing uncovers a significant performance slowdown, which could be resolved by re-configuring and re-tuning of the identified root causes during the performance test.
The BenefitsImproved system quality and performance by pinpointing performance bottlenecks and other potential problems during performance tests.Highlighted specific area of code in the application where changes might significantly improve performance.
Case Study: Test Project Overview
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
SADC Test Methodology
The overall test methodology employed by SADC for conducting the performance test is based on the Concurrent Users Load and Automated Simulation paradigm. This method is also referred to as the Automated Performance Test.
WorkloadModel
Automated testscripts f or applications ofthe SUT to beperf ormance-tested.These scripts areplay ed back during test execution,and assigned to VUs.
Applicationsyst em
components
Work-day Period
Resourcemonitoragents
Performance Test MethodologyNew Paradigm in Load Test/Stress Test: Virtual Users & Automated Simulation
Virtual Users (VUs)System processes w hichtakes the placeof actual users.
PerformanceTestServer
Performance Test CAST Tool
Generates required number of VUs, each VU executing the desired apps-functions according to the defined W orkload M odel.
Selected serversare monitored for its resource utilization. Resource UtilizationInformation ( % RAM and %CPU )relayed back to the Performance Test Server.
SUT underPerformance Test
Actual apps transactions
Response Time & Server ResourceUtilization data arecaptured, generated,and analyzed.
ApplicationUsage
Distribution
M odels full-load productionenv ironment of the SUTto be performance-tested.This is transcribed fromthe Application UsageDistribution.Workload M odel used asbasis to generate VUsand execute requiredautomated test scripts.
RelevantTest Plans,Test Procedures.
WorkloadModel
Automated testscripts f or applications ofthe SUT to beperf ormance-tested.These scripts areplay ed back during test execution,and assigned to VUs.
Applicationsyst em
components
Work-day Period
Applicationsyst em
components
Work-day Period
Resourcemonitoragents
Resourcemonitoragents
Performance Test MethodologyNew Paradigm in Load Test/Stress Test: Virtual Users & Automated Simulation
Virtual Users (VUs)System processes w hichtakes the placeof actual users.
PerformanceTestServer
Performance Test CAST Tool
Generates required number of VUs, each VU executing the desired apps-functions according to the defined W orkload M odel.
Selected serversare monitored for its resource utilization. Resource UtilizationInformation ( % RAM and %CPU )relayed back to the Performance Test Server.
SUT underPerformance Test
Actual apps transactions
Response Time & Server ResourceUtilization data arecaptured, generated,and analyzed.
ApplicationUsage
Distribution
M odels full-load productionenv ironment of the SUTto be performance-tested.This is transcribed fromthe Application UsageDistribution.Workload M odel used asbasis to generate VUsand execute requiredautomated test scripts.
RelevantTest Plans,Test Procedures.
Case Study: Test Project Overview
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
How SADC Managed the Test Project
Case Study: Test Project Overview
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Pre-Project Test
Requirement
Research/ Analysis
Test Design
Implement Design and Test Setup
Test Execution and
Test Management
Test Assessment
Test Project Closure
Test Project Phases
•Define test project requirements.•Define Testing Scope•Define Test Project Timeline Estimation•Define Test Criteria•Agree on Communication Channels•Agree on Reporting Procedures, Method and FrequencyK
ey A
ctiv
ity
Method/Tools
•Analysis of test project requirements & relevant Client A documentation.•Analysis & mobilization of test execution requirements –manpower, test tools (hardware, software).•Design test configuration, test procedures, test scenarios, test cases.•Develop Workload Distribution Model (Load-Test Model).•Develop Test Plan/Test Procedures.
•Set up of SADC Test Equipments at test site •Set up of SADC Test Tools •Test Data Preparation •Automated Performance Test Scripts Generation •Pre-Test Execution to check all the automated test scripts are functioning as per requirement.
•Execute Test Procedures/Test Scenarios.•Manage Test Execution.•Record results of test execution.•Monitor & Capture various resource utilization data.
•Analyze Test Results•Final Test Assessment Report Documentation.•Final Test Assessment Report Briefing.
•Test Environment clean-up.•Test Project close-off (archive test artifacts; documentation, test datastores)
Case Study: Test Project Phases
Output
Input
Test Project Charter
Test Project Proposal
Test Plan Test Assessment Report
Certificate of Acceptance
Test PlanTest Project Charter
Test Assessment Report
Business Requirement
Test Suites Scripts and Test-related Documents
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
Case Study: Lessons Learned
The combination of PMBOK® and CMMI in SADC’s Quality Management System (QMS) Framework and Test Methodology Framework (TMF), provides a well-structured manner in producing high quality output and managing test projects.
Challenges CommunicationEnforcementTrade-off
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
For further information, please visit us at www.mscinnovationcentre.com.my/sadc
Khor Cheng San ([email protected])
&Azdilizan Abdullah
Managing Independent Software Testing Project - A Case Study from Multimedia Super Corridor
Copyright Protected 2005 MDC
MSC Innovation Centre
FOR YOUR ATTENTION