Improving Quality and Cost-effectiveness in Enterprise Software Application Development: an Open, Holistic Approach for Project Monitoring & Control

Improving Quality and Cost-effectiveness in Enterprise Software Application Development:

an Open, Holistic Approach for Project Monitoring & Control

Ernesto Damiani, Fulvio FratiUniversità degli Studi di Milano, Italy

Luigi Buglione, Sergio Oltolina, Gabriele RuffattiEngineering Group, Italy

2nd International Conference on Software BusinessBrussels, 8-10 June 2011

ICSOB 2011 2° Int. Conference on Software BusinessBrussels (Belgium), June 8-10 2011

http://www.eng.it/

http://www.icsob.org/

Outline

• Introduction

• The Puzzle of Project Monitoring & Control

• Case study

• Conclusions & Future Works


Need for OS SPM Tools

• Open Source Software (OSS) relevance rapidly increased mainly for the OSS capability – Fostering open knowledge sharing across organizations– Reducing Total Cost of Ownership (TCO) – Increasing Return on Investment (ROI)

• A large amount of OSS is freely available covering plenty of informative and business goals– However, very few OSS projects deal with goal-oriented

measurement gathering data directly from the organization’s information systems


Our Objective

• Describe a new, more comprehensive approach to software project management

• Propose a roadmap to set up and manage a reliable and efficient measurement framework

• Present Spago4Q, a complete open source platform for process monitoring and automatic data gathering


The Puzzle of Project Monitoring & Control

• Measurement plan set-up often influenced by – Adoption of “standard measures” following an

adoption-by-analogy approach– Reduction of budget devoted to measurement

& monitoring, leading to a lower level of control on the project


– Lack or scarcity of reliable data, making it hard to bind the mean relative error to the phenomena to be predicted

• A sound analysis of informative needs, measurements, and metrics represents the main pieces of our Project Monitoring & Control framework

GQM Approach - 1• Propose a three-tier decomposition, deriving

measures (M) from the related questions (Q) to be posed for answering the information goals (G)of the interested stakeholders– www.gqm.nl

• GQM-based measurement frameworks are multi-faceted supporting multiple perspectives and analysis dimensions

• the Measurement Information Model (MIM) links the information needs to its measurable entities and related attributes– Refines and reinforces the basic GQM idea, stressing the central role of

information needs and the instrumental role of measures– Ref: ISO/IEC 15939:2007, Appendix A


http://www.gqm.nl/

Measurement Information Model (MIM)


BMP Technique

• BMP (Balancing Multiple Perspective) is a technique for determining the right number of measures to be applied within a measurement plan

• www.semq.eu/leng/modtechbmp.htm

• Characterized by four steps:1. Determine the dimensions of interest in the project

2. Determine the list of the most representative measures associated with each dimension

3. Identify which other control variables might be impacted negatively (e.g. higher quality often means greater initial costs or longer project duration);

4. Figure out the best combination of indicators to build a measurement plan for the project.


http://www.semq.eu/leng/modtechbmp.htm

The QEST-LIME Family• Performance evaluation model designed to

tackle decision-making process from a multi-perspective viewpoint– Analyses process Quality under, at least, the Economic,

Social, and Technological dimensions– Provides an open, multidimensional shell according to the

management objectives of each specific development project– Expresses performance into a unique, single value, as the combination

of the specific measures

• The QEST is extended by the LIME model to manage dynamic contexts– Collection, monitoring, and control of multidimensional measures are

applied at each software life cycle phase– http://www.semq.eu/leng/modtechqlm.htm


http://www.semq.eu/leng/modtechqlm.htm

Spago4Q• Spago4Q (SpagoBI for Quality) is

– www.spago4q.org – An OS platform for the continuous monitoring of software

quality– A vertical adaptation of OS Business Intelligence

suite SpagoBI,– Recently adopted by the European Commission –

Directorate General for Regional Policy (DG-REGIO)

• Spago4Q provides– Multi-process multi-project monitoring– Automatic data collection executed in a fully transparent way– Equipped with extractors specific of most-common software process

environments (IDE, workflow management, text editing, etc.)

• Based on a modular metamodel that allows to apply, at the same time, to running projects adopting different development process, the same measurement framework


http://www.spago4q.org/


Spago4Q Metamodel


Spago4Q Architecture


Putting all Pieces Together

• GQM, BMP, QEST-LIME, and tools like Spago4Q provide the pieces needed to solve the process monitoring puzzle

• Our approach proposes to join them, keeping the best from each – Define what we need to measure (using a GQM-based technique)– Filter and prioritize what we need to measure (using the BMP technique).– Determine the project/activity performance values with a holistic view (using QEST-

LIME), highlighting the improvement goals to manage.– Automate the last step with an OSS tool (e.g. Spago4Q), providing faster and transparent

data collecting


Case Study – 1

• The framework was applied to two large proprietary products of Engineering Health Software Factory (HSF)– projects data were analyzed and collected in the period from January

2009 to October 2010


Case Study – 2

• To monitor each phase of the HSF software life cycle, we exploit BMP and LIME

• The main business goal for our goal-oriented analysis was to reduce the overall production cost

• Such cost is mainly driven by three factors: – Lack of requirement stability, as a source of overhead in design and

implementation activities– Working group management overhead, as a source of delays and variance

in milestones– Corrective and adaptive maintenance activities

• Cost factors allowed to define the measurement goals and the complete list of measures, associated with the respective goal


Cost Factors

Requirements (analysis)

Measurements w.r.t. Cost FactorsGoal Measure

CMMI-DEV v1.3 PAs

Measure Id.

REQ-E- 1 Incidence of delays on delivery milestones (deliverables) w.r.t. total # deliverables PMC – MA PR-REQ-E-M1.1

REQ-E-2 Requirements VariabilityRD – REQM - PMC

PR-REQ-E-M1.2

REQ-S-1 #. detected criticalities during the human resources management w.r.t. group size in the phase PPQA – PMC PR-REQ-S-M1.1

REQ-S- 2 User satisfaction or % users involvement in the phase all PA (GP2.7 PR-REQ-S-M1.2

REQ-T- 1 Document quality: respect of quality standard PPQA PR-REQ-T-M1.1

DEV-E- 1 Incidence of delays on delivery milestones (deliverables) w.r.t. total # deliverables PMC – MA PR-DEV-E-M2.1

DEV-S- 1 # detected criticalities during the human resources management w.r.t. group size in the phase PPQA – PMC PR-DEV-S-M2.1

DEV-T- 1 Document quality: respect of quality standard PPQA PR-DEV-T-M2.1

DEV-T- 2 Software quality: complexity, compliance, maintainability PPQA – VAL PR-DEV-T-M2.2

DEV-T- 3 Compliance with end-phase checklist PPQA PR-DEV-T-M2.3

TES-E- 1 # Defects detected before System Test by PR or verifications w.r.t. code size VER – VA – PMC PR-TES-E-M3.1

TES-E- 2 Incidence of delays on delivery milestones (deliverables) w.r.t. total # deliverables PMC – MA PR-TES-E-M3.2

TES-S-1 N.o. detected criticalities during the human resources management w.r.t. group size in the phase PPQA – PMC PR-TES-S-M3.1

TES-T- 1Incidence of the n.o. reviews (peer reviews or inspection reviews) w.r.t. total # deliverables PPQA – VER PR-TES-T-M3.1

Percentage defects distribution on phases that produced the defects (consider only analysis and design phase) PP – PMC – MA PR-TES-T-M3.2

TES-T- 2 Compliance with end-phase checklist PPQA – PMC PR-TES-T-M3.3

MAN-E- 1Incidence of defects tested in running and testing phase w.r.t. maintained code size (Lines of Code or Function Points)

PMC – MA PR-MAN-E-M4.1

Mean defect resolution time w.r.t. severity during running phase IRP PR-MAN-E-M4.2

MAN-S- 1 N.o. detected criticalities during the human resources management w.r.t. group size in the phase PPQA – PMC PR-MAN-S-M4.1

MAN-S- 2 User satisfaction all PA (GP2.7) PR-MAN-S-M4.2

MAN-T-1 Percentage defects distribution on phases that produced the defects (consider only analysis and design phase) PP – PMC - MA PR-MAN-T-M4.1

Results analysis - 1


• The analysis of measures results suggests different improvement actions to monitor each phase, to improve the overall economic results– Increase document quality produced in the requirements

phase (PR-REQ-T-M1.1)– Increase software quality to facilitate its maintainability in

the development phase (PR-DEV-T-M2.1 and PR-DEV-T-M2.2)

– Increase the number of reviewed deliverables (PR-TES-T-M3.1)

• The reviews performed in the analysis and design phases had the goal to discover bugs early in the development cycle (PT-MAN-T-M4.1, PR-TES-T-M3.2)

Results analysis - 2 Product 1 Product 2

Goals Measure \ Year 2009 2010 2009 2010

Reduce Defects resolution costsEconomic DimensionPR-MAN-E

0.7000 0.7550 0.7870 0.8530

Reduce criticality in working group management

PR-MAN-E-M4.1 0.5244 0.5099 0.5793 0.7319PR-MAN-E-M4.2 0.8173 0.9244 0.9262 0.9333Social DimensionPR-MAN-S

0.6810 0.7000 0.7100 0.6850

PR-MAN-S-M4.1 0.6944 0.5833 0.5000 0.3000

User satisfactionPR-MAN-S-M4.1 0.6750 0.7500 0.8000 0.8500Technical DimensionPR-MAN-T

0.6304 0.6552 0.7005 0.6842

Increased #. of checks/reviews PR-MAN-T-M4.1 0.6304 0.6552 0.7005 0.6842


• Analysis of the goal Reduce Defects Resolution Cost in the Running phase– Improvement of the 7.2% for Product 1 and 7.6% for Product 2– Improvement of Mean Defect Resolution Time measure (PR-MAN-E-

M4.2) – Consequent improvement of Social (S) dimension, and in particular the

User Satisfaction metric (PR-MAN-S-M4.2)• Received benefit from the improvements of document quality and the reduction

of Mean Defect Resolution Time

Results Analysis – 3 Product 1 Product 2

Goals Measure \ Year 2009 2010 2009 2010

Compliance with pre-defined documents q.l. Requirement phasePR-REQ-T-M1.1

0.8333 0.8667 0.8667 0.9000

Compliance with pre-defined documents q.l. Development phasePR-DEV-T-M2.1

0.8333 0.8667 0.8667 0.9000

Compliance with pre-defined software q.l. Development phasePR-DEV-T-M2.2

0.5000 0.6000 0.7500 0.9000

Increase n.o. checks/reviewsTest phasePR-TES-T-M3.1

0.7500 0.8750 0.5250 0.6250


• Analysis of measures correlated to the Reduce Defects Resolution Cost goal– The increasing of the number of reviews (PR-TES-T-M3.1) had a

positive effect on the improvement of the overall quality of documents (PR-REQ-T-M1.1 and PR-DEV-T-M2.1)

– The improvement of software quality (PR-DEV-T-M2.2 w.r.t. PR-MAN-T-M4.1) was mainly due to the decreasing the percentage of defects in initial development phases (analysis and design)

Results Analysis – 4


Conclusions

• We present our experience in developing an integrated framework for process monitoring integrating– GQM approach– BPM technique– QEST-LIME framework– Spago4Q platform

• The effectiveness of the framework has been tested on two on-going projects, and an analysis of the achieved results has been resented

• Our approach represents a good starting point for a full implementation of a Balanced Scorecard (BSC) technique


Future Works

• Future works will focus on – the improvement of Spago4Q reporting features – implementation of a GQM(R) matrix for choosing new

possible measures in order to cover a larger plateau of information needs at the same cost

• The framework is available as a demo on the Spago4Q web site (www.spago4q.org)



Questions?

THANKS FOR YOUR ATTENTION!THANKS FOR YOUR ATTENTION!

Contacts:

http://sesar.dti.unimi.it http://www.spagoworld.org

[email protected]
