OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Metricsin the context of the CMM/SPICE
SPIN-UK, 29 September 1998
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Some views on measurement…• “What is not measurable, make measurable.”- Galileo• “When you can measure what you are speaking about and express it in
numbers you know something about it; but when you cannot measure it, when you cannot express it in numbers your knowledge is of a meagre and unsatisfactory kind.” - Lord Kelvin
• “…This remark [above] has been quoted with an approval it does not altogether deserve - it does not, for example, do justice to the work of Darwin or Pasteur, to name only two of Kelvin’s contemporaries.” - G.U.Yule (statistician)
• “You can’t control what you can’t measure.” - DeMarco• “Metrics is crap.” - Dijkstra
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
What is measurement?
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Empirical relational system
real world
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Empirical relational system
Formal relational systemmeasurement
real world mathematical world
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Empirical relational system
Formal relational system
Results
measurement
mathematics and
statistics
real world mathematical world
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Empirical relational system
Relevantempirical
information
Formal relational system
Results
measurement
interpretation
mathematics and
statistics
real world mathematical world
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Empirical relational system
Relevantempirical
information
Formal relational system
Results
measurement
interpretation
mathematics and
statistics
decisions and
actions
real world mathematical world
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Characteristics...
• good measurement should not (in general) conflict with empirical insights
• sound measurement systems reflect the level of understanding of those things that are measured
• good measurement encourages improved understanding
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Empirical relational system
Relevantempirical
information
Formal relational system
Results
refinedmeasurement
more
sophisticated
interpretation
mathematics and
statistics
betterdecisions
and actions
real world mathematical world
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Measurement - state of practice…
• can be self serving, but increasing awareness of role
• often not well founded, but effort is invested
• data collected but often not valued (even when useful)
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Effective measurement require an appreciation of...
• measurement fundamentals– how do we measure?
• context– what do we need to measure?
– what can we measure?
• soft (human) factors– why should I measure/be measured?
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Measurement within the Capability Maturity Model:
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Initial
Repeatable
Defined
Managed
OptimizingThe Capability Maturity Model
Organizational abilities are described as five levelsof increasing capability
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Defined
maturity level key process areas
activities
commitment
abilityverification
M & A
common features
measurement and analysis
in each KPA from levels 2 to 5
ISM
TP
PR
OPD
IGC
SPE
OPF
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Measurement within the CMM...
• is a common feature of every key process area at each described maturity level
• measurement and analysis common feature is least well described– SEI recognize need for organizations to derive their own measures– SEI say nothing about how to measure
• in practice we tend to find measurement and analysis is the last common feature to be satisfied for any given KPA– can use this as an informal assessment ‘short cut’.
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
...Measurement within the CMM...
• When investigating measurement capability look for...
– a complementary view of quality throughout the organization
– recognition of why data is collected, and understanding of definitions and analysis
– data verification (accuracy known) and measures validated
– information used to make decisions and direct actions
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
...Measurement within the CMM
• Level 4
– measure products and process quantitatively• Software Quality Management• Quantitative Process Management
– aspires to production engineering (cf project engineering)
– still controversial
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Measurement within SPICE (ISO 15504)
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
The SPICE Architecture
ORG
ENG
SUPP
PROJ
CUS
Proc 4
Proc 2Proc 1
Proc 4
Eng. 3Develop S/W Design
Proc 554
210
Practice 1
Practice 2
Practice 3Eng 3.4 Establish
traceability
ProcessCapability
Welldefined
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
A comparison of the CMM and SPICE
• SPICE...
– is more complex• this is a good thing and a bad thing
– it may be unrealistic– but does require thought
– is structured by process• ...not capability - so may make arbitrary target setting more difficult• allows all processes to be addressed to some extent (c.f. CMM’s SPE)
– has process capability defined by generic processes
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Measurement within SPICE...
• Measures required throughout (like CMM)...
– Generic Practices (define process capability of every process)
• level 2 - plan process (I.e. estimation)
• level 2 - track with measurement
– note - “use of measurement implies measures have been defined and selected and data has been collected”
• level 3 - use well defined data
• level 4 - measurement oriented (like CMM level 4)
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
...Measurement within SPICE...
• ...Measures required throughout (like CMM)
– Processes
• engineering 2.2 hints at ISO9126
• project 2.5 and 7.2 - size, effort,cost schedule resources risks
• project 5.2 - define quality metrics
• organization 2.9 - define process measures
• organization 2.12 - set performance expectations
• organization 3.6 - define measures of impact
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
...Measurement within SPICE
• introduced at low capability level (like CMM)
• states expectation of what measurement requires (unlike CMM)– (even at low capability level)
• minimal guidance (examples) on what to measure (less than CMM)
• implies a distinct capability to define and assess measures
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
Summary
• Measurement
• within the Capability Maturity Model
• within SPICE (ISO 15504)
OXFORD SOFTWARE ENGINEERINGSoftware Engineering Services & Consultancy
C.C.Shelley
[email protected]+44 (0) 1993 700878
O X F O R D
S O F T W A R E E N G I N E E R I N G
L I M I T E D
www.osel.co.uk