Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt 1
Define
Parent Project Title:
QA Process,Standards & Automation introduced to improve Development of websites and web-services ( Black Belt) Project By Ms Meenakshi Ganeriwala .
Start Date: 01/07/2009
Estimated End Date: 01/06/2010
Actual End Date:
Project Scope:
IS IS NOT
Critical to ACE Infoway production
Client support center of ACE Infoway
Critical to ACE Management Team
Mock ups for Clients
Critical to ACE Development Team
Data entry sector of ACE Infoway
Critical to ACE Design Team Does not involve creativity for designs
By Ms Meenakshi Ganeriwala
2
Related Projects
1. Yellow Belt by Ms Meenakshi Ganeriwala ACE Infoway Pvt Lmt
2. Black Belt Project by Mr Sandeep Kulkarni , Sr. Manager @DFS Austin Texas USA
3. Black Belt by Vision Raval
Key Learnings;
4. Development inaccuracy and instability with higher loads and varied technologies like PHP, Joomla, ,NET, Java etc
5. Absence of proper QA cycle reduces market value leading to loss of revenue. ( This is captured from one of the meetings with Dr Anu . He referred to Clients who deducted amount from payments due to increased no of defects existing in the deliverables)
6. Proactive vs. reactive monitoring of Client Deliverables
7. Data analysis and statistical significance of data from reports and charts
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Define
By Ms Meenakshi Ganeriwala
3Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Team:
Project Lead :
Ms Meenakshi Ganeriwala ACE Infoway QA and R &D Manager
Project ID: 2426250
Team Members :
Kirtan Gajjar ACE Infoway Sr QA Analyst
Manish Varsani ACE Infoway Manager QA
Rushabh Patel ACE Infoway Jr QA Analyst
Shital Chavda ACE Infoway QA Analalyst
Exec Sponsor :
Dr P M Anarvartham ACE Infoway Vice President
BPI Mentor :
Sandeep Kulkarni DFS BPI Area Champion USA
Vision Raval Vision Enhancer
Define
By Ms Meenakshi Ganeriwala
Roles and Responsibilities
Manish : Execute all QA process laid out in Ahmedabad in Rajkot for QA
Kirtan : Regular updates to Blog, Status report Updates PAS allocation, All QA activities
Research on New QA Tools
Rushab: Updating Defect reports, Status report Updates
PAS allocation, All QA activities
Research on New QA Tools
4
Project Definition
Setting up Quality Assurance Center to enable quality testing of ACE applications , improving the existing level of “timely response” of development and quality of deliverables to the Clients, to meet the SLA of Clients.
Business Reason
The number and various types of applications under development in Ace Infoway is increasing rapidly . This is resulting in the increase of delivery issues for both quality and SLA agreement. This project of setting up a department which incorporated both Quality Assurance and Research was initiated to support the different technology to be used for QA and to scale up the performance of development and deliverables at ACE Infoway and there in work to resolve /eliminate the evident causes of development failures and issues. This would eliminate / reduce Clients insistence to alter pricing after delivery.
Desired State
All End to End applications to under go QA cycle with usage of all QA Tools and Standards declared at ACE Infoway, while improving application development by reduction in the number of defects of applications. To train all resources in the QA Tools and standard followed at ACE Infoway Pvt Lmt.
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Define
By Ms Meenakshi Ganeriwala
5
Customer Output Process Input Supplier
Design Department
QA comparison with Client scope
Design QA process
Design HTML pages
Project Manager /Design
HeadDevelopment Mantis /Test
cases/QA feedback Document
Running QA using Mantis,
Developed Code, Client scope,
technical details of Development environment
Project Manager/Develop
er
Development Code review /Functionality
review
Root cause analysis and automation
CSS details Development
Production /Client
Optimized Code/Documents
Tested Code and Scope
Design/Code Documents
QA Department
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
COPIS
Define
By Ms Meenakshi Ganeriwala
6Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Define
By Ms Meenakshi Ganeriwala
7
Critical to Quality Tree
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Project Management Defect Free Design
Fewer issues Design
No Of design defects W3C
validation
Customer Need Drivers CTQs (Metric)
Defect free Code Timely response to application issues
No Of defects Use of QA Tools
End ClientsStandard and
Optimized code delivery
Reduce test time QA Automation
Xenu, Sniffer, QTP
Reduce Development
Resource
Reuse code from repository
Reusable Code and Design SLA compliance
Timely response to application issues
Define
By Ms Meenakshi Ganeriwala
CTQ - Critical to Quality (CTQ) variablesThis is done in the define stage
8
Estimated Project Cost Benefits
1. With the introduction of QA as a separate Dept 2. With using Trained QA professional3. Training developers to utilize QA Tools 4. Dedicated QA environment5. Using automation in QA 6. Submit Test Cases to Clients
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
MeasureBy Ms Meenakshi Ganeriwala
COST BENEFIT ANAYSIS
DescriptorUnit Of
MeasureFamily of Measure
ObjectiveEstimated
ChangeSavings Typein RS Showed Improvement
Number Of QA resource Cont Financial Lowered 50% 600,000 YES
NO of QA hours savings calculated per 100 hours lowered
% Financial Lowered 50% 1,00,000
YES
No of defects % Quality Improved 30%Reflection of QA hours
reduced YES
Number of QA ed Applications Count Productivity increased 100% soft
YES
SLA complience NA Quality improved 20% soft
YES
Number of Resources trained in QA Tools
% Financial improved 60% 50,000
YES
Total 750,000
Project Costs 50000
Net savings 7,00,000
9Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Measure
By Ms Meenakshi Ganeriwala
Estimated Project Cost BenefitsSupporting slide for # 8
QA started with MBG, Hemant, Raxit, Hemlata, Kirtan, Shital , Jignesh partially /replaced by manish
No of resources
Average salary /month
% improvement
Period
7 1,00,000 Pre- QA Automation
July- Dec 09
4 50,000 50 % Post QA Automation
Jan- May 10
Now QA Has MBG, Kirtan (shared), Rushab 9shared),Shital, Manish for R&D
10
Measurement System Evaluation
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Before Sample Size / Plan
Defects from the Defect reports: (before application of Automation of QA dept ) –
One set of defect reports would show defects before application of Automation and standards while another set would reflect the data after the application of automated tools and standards for QA.
Status reports (prior to QA Automation) – Quarterly data was used from the status report. It maintains the status of a particular project once it enters QA
Applications – Number of Applications supported is captured from Sheet sent weekly by Beta and reports collected weekly from all PMs (Central Application Repository)
Resources – Number of development Resources is calculated from headcount reports.
Sampling Plan to ensure random and unbiased sampling, data was used from production and development
Data Collection – For calculation of defects data was extracted from three reports 1. Defect Reports 2. Status report 3. Delay report
An MSE was performed on defects that were recorded when QA dept was absent and no automation in place.The metric is an accurate reflection of the end – user /Client’s issues with the delivered website or related web services.
Measure
By Ms Meenakshi Ganeriwala
11
Measure
A Sample defect report Data collection from Sep 2009 till May 2010
By Ms Meenakshi Ganeriwala
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
12
Measure
Sample data was collected for a period of time From Sept to Dec 2009 and defects Were ploted against 100 spent hours
This needs improvement . Peaks need to drop indication lower no of defects
By Ms Meenakshi Ganeriwala
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
13
PROCESS FLOW ANALYSIS
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Analyze
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
14
Analyze
Process Map Analysis
No. QA Standards like Checklist
No design and development standards
No QA Automation
No QA scheduling
No proper allocation and calculation of QA hours and work load
No Unbiased separate QA for Client deliverables
Fishbone Analysis
Too many defects
No operational reporting/review of metrics
Insufficient knowledge of deliverables
Installs to production being done with out QA
By Ms Meenakshi Ganeriwala
15
Analyze
Pareto analysis reveals top contributors to the defects
5 WHY’s Analysis
Why is the validations having great no of defects?- Because not one standard or checklist is being followed
Why is code and functionality having issues That is programming ?- Because not one standard or checklist is being followed
Why is the root cause of large no of defects ,not being addressed ?- Because there was no analysis being done to address defective design and coding.
Why was no analysis being performed ?- Because there was no separate unbiased group to perform QA
Why was there no department of QA ?- The process that included separate QA was not present.
WHY
Pareto Chart
By Ms Meenakshi Ganeriwala
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
WHY
WHY
WHY
WHY
The highest contributors are •Defects from Validations•Defects from Functionality•Defects from Code programming
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt 16
Improve
Series of brainstorming sessions assisted the team to gather all possible solutions and to group them into main improvement headings.
Areas of Opportunity
Possible Solutions were generated through the use of Brainstorming sessions
By Ms Meenakshi Ganeriwala
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt 17
Improve
Risks:Of the solutions selected, one needed further risk analysis. The creation of a dedicated team /Automation Tools
could lead to disruption hence we needed to ascertain the risks prior to implementation.A force field was chosen over an FMEA, ( Failure Mode Effect Analysis) Risk could be assessed easier/quicker
Key Risk Actions derived from Force Field Analysis
Involvement of Top management to propagate the existence of QA and its importance
Met and Brainstorming with individual PMs and their Teams to understand QA and its standards
Arranging training of developers on QA standards
By Ms Meenakshi Ganeriwala
One of the initial steps was to established a dedicated and Skilled QA Team and to support usage of Automation and standards a R &D Team was set up.
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt 18
QA Manager
QA Operations Head
Segment Lead 1 Segment Lead 2
Design QA Resource :
: n
Code QA Resource :
: n
Design QA Resource :
: n
Code QA Resource :
: n
QA R&D Team
Research Resource
Improve Phase -I
By Ms Meenakshi Ganeriwala
Established a Dedicated QA Environment
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt 19
Location 1. QA server Details1.Operating System : (Linux)Fadora core 7.52.Space 40 GB3.Xampp 1.7 Location 2.
QA Server Details:1.Operating System (Linux)Fadora Core 7.52.Xampp 1.73.Space 160 GB
Improve
We are using XAMPP 1.7 for the QA server including Apche 2.0,Mysql 5.1.3,PHP 5.2.8,Perl, Open SSl and phpmyadmin. XAMPP is a free and open source cross-platform web server package, consisting mainly of the Apache HTTP Server, MySQL database, and interpreters for scripts written in the PHP and Perl programming languages.
By Ms Meenakshi Ganeriwala
Creation of QA Environment
Tuesday, April 18, 2023SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
20
Sample DataHere the number like 675, 689 are project numbers.
Server
Project Number
Site
QA_feedback
date
QA (project no) (date)-ver.doc
DEV (project no) (date)-ver.docDate format (DD-MM-YY). ver is
for versioning e.g. 1,2,3. (This is QA feedback reply from development)
Date Format (DDMMMYY based on version
e.g. 09sep09, 17oct09
Complete Site (code) based onversion e.g. site, site1, site2
Date format (DD-MM-YY)ver is for versioning e.g. 1,2,3
(This is QA feedback)
Server Description: LinuxAhm : http://192.168.10.222/Qctest/
Rjt : http://192.168.52.9/
Documents related to the sitee.g. Site configuration, Requirement
Specification, database, etc.,
Directory Structure Notes
Docs
ImproveBy Ms Meenakshi Ganeriwala
21Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Improve Phase- IBy Ms Meenakshi Ganeriwala
Enhance the Team for QA and R & D
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt 22
QA Manager
QA Operations Head
Segment Lead 1 Segment Lead 2
Design QA Resource
… n
Code ReviewQA Resource… n
Design QA Resource…. n
Code ReviewQA Resource….n
R&D Team
Research Resource
ProductionSupport Lead
L1 Production Support……n
L2 Production Support…n
Improve Phase - IIBy Ms Meenakshi Ganeriwala
23Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Improve Phase - IIBy Ms Meenakshi Ganeriwala
24
Process Diagram PAS Hours
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
This enables exact calculation of PAS hours ( billable hours for QA Process.
Improve
PAS hours utilization for example resource
PAS hours of every QA resource id closely monitored to have accurate Billable hours.
Attempt is made to make sure the process defined is cost effective
By Ms Meenakshi Ganeriwala
25Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Improve – Phase iii
By Ms Meenakshi Ganeriwala
This Phase of QA is specially designed to handle errors occurring in the production environment due to non functioning of the sql statements embeded in the code that fetch data from the database. For example usage of count (*) instead of designing a sql statement that fetches only required fields.
26
Improve
After implementation of the new QA standards and Tools the Following situation was observedFewer no of defects for projects per 100 spent hours
By Ms Meenakshi Ganeriwala
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Defects do not run more than 35 numbers
Tools used•Xenu •Zend Studio •Dead Links•Mantis•QTP•Load Runner•WinRunner
Improved range
RANGE
27
Improve
Comparison chart of data pre and post implementation of QA standards
By Ms Meenakshi Ganeriwala
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
The diagram and the spread sheet show clear improvement in defect detection and solution after the implementation of QA standards, Tools and Training to resources ( The PMs and Developers)
28
Before Implementation of QA Standards and Automaton
After Implementation of QA Standards and Automation
Improve
By Ms Meenakshi Ganeriwala
Tuesday, April 18, 2023SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
29
Improve
By Ms Meenakshi Ganeriwala
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
This slide is to support Slide No 30 and Slide 8, % improvement in the QA hours Utilized is Sampled out for reference
For every 100 $ spent in QA there is a reduction of approx 50 % in the cost that is rounded Rs 53620 for period six months (Oct 2009 – March2010 )year savings should be Total QA hours --- 3574 hours as per spread sheetTotal savings yearly --- 1,00,000 RsReference slides ----- PAS hours data, TITO data
30
Improve
By Ms Meenakshi Ganeriwala
AREARS OF IMPROVEMENT1. No actual hours of QA accounted2. No use of QA Automation3. PAS not noted
IMPROVED 1. Gain in $ amount showed2. Actual data recorded3. PAS monitored4. QA Automation used
FROM
July 09 to Dec
0909
FROM Jan 10 to May10
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Improvement seen by 50 %
31
ImproveArea of improvement detected in the Analyze Phase
By Ms Meenakshi Ganeriwala
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
SIGMA IS IMPROVED from 54.46 to 8.28Process improvements resulted insignificant reduction in Mean & STD!
32
ImproveBy Ms Meenakshi Ganeriwala
Tuesday, April 18, 2023SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Points are more centered and the same can be observed in the above diagram
Before Implementation of QA Standards and Automaton
After Implementation of QA Standards and Automation
Test for Normality
(Weekly Defects reports - after)
33
ImproveBy Ms Meenakshi Ganeriwala
Tuesday, April 18, 2023SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Cp (before) = 0.0058, Cp (after) = 0.0832 ( Will it ? ) Improved potential capability by reducing variation
Cpk (before) = 0.000581, Cpk (after) = 0.38245 ( Does it ?) The new process is several times more effective at producing results within the
specification because variation was reduced and new process is centered close to target.
34
QA Scheduling
Checklist1. Checklist for design 2. Checklist for
development.3. Coding Standards as per
document in ACE
Automation Tools4. W3C validation5. Zend Studio6. Dead link7. Xenu8. PHP Codesniffer9. Mantis Bug Tracker
( referance to slide 24 )10. QTP ( Quick Test
Professional by HP )
Tuesday, April 18, 2023SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
1. Procurement: QA team receives the data of new projects (From Project Manager/ Beeta )
2. PM send report to QA Lead every Saturday for scheduling of projects
Scheduling: Based on the size of the project and go live date, the project is scheduled for QA
3. Control:QA notifies by email to Project Manager for:1) The required documents supporting project requirement to QA2) Website configuration setting to be done while uploading code on
QA server3) Implementing QA feedback4) Rescheduling QA if requested for any project by project manager5) Completion of QA
Control
QA Standards and Tools
By Ms Meenakshi Ganeriwala
35
QA Reports
1. QA tracks the schedule for the project Document name: QA_schedule.xls QA schedule <dd-mm-yyyy>Link: \\ace-iis\QA-Doc\QA schedule
2. QA tracks the status for the project Document name: QA_status.xlsLink: \\ace-iis\QA-Doc\QA Status
3. QA provides feedback for the defects in the project.
Document name: QA_(Project number) (date).docLink: \\ace-iis\QA-Doc\QA Feedback
4. Representation of delays Delay ReportDocument name: Delay_Report_(location)_.xlsLink: \\ace-iis\QA-Doc\QA Delay Report
5. Representation of defects Defect ReportDocument name: Defect_consolidated_(location)_ report.xlsLink: \\ace-iis\QA-Doc\QA Defect Report
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Control
A sample Action log
By Ms Meenakshi Ganeriwala
36
A Sample delay Report :( A report which states why a particular project got delayed in entering the QA cycle.)
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Control
A list of QA Documents and their available updates
By Ms Meenakshi Ganeriwala
37
A sample Mantis Report as a part of QA Reporting
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Control
By Ms Meenakshi Ganeriwala
38
A sample Status Report to track the status of the project period: from Sep 2009 till May 2010
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Control
By Ms Meenakshi Ganeriwala
39Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Control
A list of QA Documents available at a particular locationRegular updates are published 1. Blog2. News Bulletin3. Send via Email for selective crowed
By Ms Meenakshi Ganeriwala
40
QA Document Repository
QA team maintains the blog highlighting the standard practices to improve the development and reduce the development time.
– Link http://192.168.10.222/developmentace/
Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
Vision DocumentLink: \\ace-iis\QA-Doc\QA Vision
Action logLink: \\ace-iis\QA-Doc\QA Action Log
PHP standardsLink: \\ace-iis\QA-Doc\QA PHP Coding Standards
Test CasesLink: \\ace-iis\QA-Doc\QA Test Cases
QA Training for resourcesLink: \\ace-iis\QA-Doc\QA_Training Doc
Knowledge Sharing @ ACE
Control
By Ms Meenakshi Ganeriwala
41Tuesday, April 18, 2023 SIX SIGMA BLACK BELT Ace Infoway Pvt Lmt
QA Research & DevelopmentPHP Code sniffer
Link http://pear.php.net/package/PHP_CodeSniffer PHP Documenter
– Link http://phpdoc.org/MANTIS
– Link http://zerozone.com/mantisbtAdmin Tool for ACE ( out of the Box)
– Link http://192.168.10.222/Qctest/admintool/adminDead Link Tool that detects dead links
– Link http://www.dead-links.com/– Link http://home.snafu.de/tilman/xenulink.html
Zend Studio – Link http://www.dead-links.com/
QTP ( Quick Test Professional by HP)– Link http://www.hp.com/
LoadRunner– Link http://www.hp.com/
WinRunner– Link http://www.hp.com/
Control
THANKS
By Ms Meenakshi Ganeriwala