Upload
techwellpresentations
View
49
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Lee Copeland maintains that most organizations have some kind of metrics program—and almost all are ineffective. After explaining the concept of measurement, Lee describes two key reasons for these almost universal metrics program failures. The first major mistake people make is forgetting that the model we are using for measurement is not necessarily reality. The second major blunder is treating ideas as if they were real things and then counting them. Lee describes the “Three Don'ts of Metrics”—Don’t measure it unless you know what it means; Don’t measure it if you’re not going to do something with the measurement; and no matter what else you do, Don’t turn your measurement into a goal. Through the years, Lee has discovered his favorite project indicator is not a measurement at all—and you’ll be surprised to learn what it is. Join Lee as he shares his Zeroth Law of Metrics to guide your program to success.
Citation preview
4/16/2013
1
1
Lee CopelandSoftware Quality Engineering
The Mismeasure of Software
The Last Talk on Measurement You’ll Ever Need to Hear
22
Topics
• What is measurement?
• The “Four Don’ts” of measurement
• My favorite indicator—isn’t a measurement
• Lee’s Zeroth Law of Measurement
4/16/2013
2
33
What Is Measurement?
• “Measurement is the empirical, objective assignment of numbers, according to a rule derived from a model or theory, to attributes of objects or events with the intent of describing them”
–Cem Kaner
44
A Classic Example
• Phrenology
– The brain is the organ of the mind
– The brain is not a homogenous entity, but an aggregate of “mental organs” with specific functions
– These organs are topographically localized
4/16/2013
3
55
A Classic Example
• Phrenology
– The relative size of any particular mental organ indicates the power or strength of that organ
– External measurement can be used to determine the internal states of our mental powers
66
A Volunteer From The Audience?
4/16/2013
4
77
A Classic Example
• Even automated devices for phrenological analysis were created. The psychographnetted its inventors about $200,000 at the 1934 Century of Progress Exposition in Chicago
• Some employers even required phrenological certificationsbefore hiring employees
88
What Is Measurement?
• Did you notice that Kaner’s definition only requires intent. It says nothing about the measurement actually being useful?
4/16/2013
5
99
Two Causes of Measurement Failure
• Map-territory error
• Reification error
1010
What Is Measurement?
• “Measurement is the empirical, objective assignment of numbers, according to a rule
derived from a model or theory, to attributes
of objects or events with the intent of describing them”
–Cem Kaner
4/16/2013
6
1111
Map-Territory Error
• A common error is the confusion of a model with reality
• “The map is not the territory”
– Alfred Korzybski, 1931
• Mathematical or simulation models may helpus understand a system or situation but real life always differs from the model
1212
Map-Territory Error
• The model / theory / map may just be WRONG!
4/16/2013
7
1313
What Is Measurement?
• “Measurement is the empirical, objective assignment of numbers, according to a rule derived from a model or theory, to attributes of
objects or events with the intent of
describing them”
–Cem Kaner
1414
Reification Error
• A Reification error occurs when an abstraction (abstract belief or hypothetical construct) is treated as if it were a physical object or a real event
• Counting them is dangerous! Comparing the counts is even more dangerous!!!
4/16/2013
8
1515
Reification Error
• Examples include:
– Intelligence
– Performance
– Commitment
– Motivation
– Leadership
– Attitude
1616
Reification Error
• Now do you understand why you feel so violated after your Annual Performance Review?
4/16/2013
9
1717
But Wait, There’s More !
• Consider these reification errors:
– Requirements
– Test cases
– Defects
1818
Topics
• What is measurement?
• The “Four Don’ts” of measurement
• My favorite indicator—isn’t a measurement
• Lee’s Zeroth Law of Measurement
4/16/2013
10
1919
The Four Don’ts
Don’t #1
Don’t #3
Don’t #2
Don’t #4
2020
Don’t #1
• Don’t measure it if you don’t know what it means
4/16/2013
11
2121
Don’t #1
• Does the number actually represent the attribute we are trying to measure?
• Examples:
• Lines of code – program size
• Number of test cases – thoroughness
• Number of defects found – product quality
• Superclass:Subclass ratio – good systemdesign
2222
Don’t #2
Thanks for all your
hard work producing
this measurement
report
• Don’t measure it if you’re not going to do something because of the measurement
4/16/2013
12
2323
Don’t #2
• What does this tell your staff?
– You’re a clown
– You’re an idiot
– You’re an H
2424
Don’t #3
• Don’t turn your measurement into a goal
• Goodhart’s Law — Proposed in 1975 by Charles Goodhart—a former advisor to the Bank of England and Emeritus Professor at the London School of Economics
4/16/2013
13
2525
Don’t #3
• The law states that “once an indicator is made a target for the purpose of guiding policy, then it will lose the information content that originally made it useful”
• Goodhart drones on, “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”
2626
Don’t #3
• Professor Marilyn Strathern has restated Goodhart's Law more succinctly and more generally:
“When a measure becomes a target, it ceases to be a good measure”
4/16/2013
14
2727
Don’t #3
• A classic metric of this type is “lines of code written per day.” When used as a measure, this metric can be valuable for estimating, planning, and tracking
• However, when LOC/day becomes a goal, developers may be enticed to write more lines of (less efficient) code
2828
Don’t #3
For example, a classic Java “for” loop has the generic form:
for (initialization;
termination; increment) {
statement(s)
}
An example of this loop is:
for (int i=1; i<11; i++) {System.out.println("Count is: " + i);}
4/16/2013
15
2929
Don’t #3
If writing more lines of code is rewarded, I can write the equivalent code as:
int i;i=1; while (i<11)
{System.out.println("Count is: " + i);i++;}
thus increasing the LOC count while providing the identical functionality
3030
Don’t #3
• Goodhart’s Law reminds us that when rewards and punishments are connected to the achievement of specific goals, unintended consequences are often the result
4/16/2013
16
3131
Don’t #3
• If LOC is too “old-school”, consider Velocity, a premier agile metric
• What have you seen happening with Velocity?
Goal
3232
Don’t #4
• Don’t focus on measuring effort– Number of test cases created– Number of test cases executed– Number of test cases passed– H
4/16/2013
17
3333
Don’t #4
• Instead, focus on measuring accomplishments– Number of defects discovered before
installation– Costs saved through testing– H
3434
Topics
• What is measurement?
• The “Four Don’ts” of measurement
• My favorite indicator—isn’t a measurement
• Lee’s Zeroth Law of Measurement
4/16/2013
18
3535
My Favorite Indicator
• Lines of code
• Percent test cases passed
• Defect density
• Defect detection percentage
3636
My Favorite Indicator
• My favorite indicator is a
4/16/2013
19
3737
My Favorite Indicator
is
3838
My Favorite Indicator Is
• Surprise
– “To discover suddenly and unexpectedly, without warning”
– “To become aware of something not previously perceived”
– “A spontaneous awareness of a discontinuity in our understanding”
4/16/2013
20
3939
My Favorite Indicator Is
• Surprise is an interesting indicator for use in software development
• We could examine the surprise itself
• We could examine the reason(s) for our surprise
4040
Satir Interaction Model
Virginia Satir
4/16/2013
21
4141
Intake
• Intake is the process of observation, of bringing part of the “world out there” into our consciousness
4242
Intake Surprises
• As humans, we fail to observe huge amounts of information; there is simply too much
• Some individuals and organizations maintain an “oblivious culture” – they chose not to systematically observe anything
4/16/2013
22
4343
Intake Surprises
• A second type observes, but quickly filters out data that does not match their world-view
• A third, to prevent having to deal with realities, actually prohibits observing
4444
Meaning
• Meaning is the process of mapping our observations to our context, experience, and feelings
• Biases, agendas, pressures, expectations, and even past unresolved personal issuescan cloud our assignment of meaning
4/16/2013
23
4545
Meaning Surprises
• If not careful, we may assume that the first meaning we assign is the correct meaning is the only meaning
• Remember Weinberg’s Rule of Three – “if you can’t think of at least three different meanings, you haven’t thought enough about it”
4646
Significance
• Significance is the process of deciding whether the observation and meaning are important enough to affect our actions
4/16/2013
24
4747
Significance Surprises
• We may fail to assign the proper significance because:
– the observation does not fit into our previous experience
– we just can’t evaluate its importance
– we may be operating under rules that do not serve us well
– we may have been conditioned to minimize the significance
4848
Significance Surprises
4/16/2013
25
4949
Surprise
• Focus on your processes of Intake, Meaning, and Significance
• To learn something, examine the surprise
• To learn even more, examine the reasons why you were surprised
5050
Topics
• What is measurement?
• The “Four Don’ts” of measurement
• My favorite indicator—isn’t a measurement
• Lee’s Zeroth Law of Measurement
4/16/2013
26
5151
Lee’s Zeroth Law of Measurement
5252
Lee’s Zeroth Law of Measurement
• “If you measure the wrong thing, and reward the wrong thing, don’t be surprised if you get the wrong thing”
FamousGuru
&Grandpa
4/16/2013
27
5353
Recap
• What is measurement?
• The “Four Don’ts” of measurement
• My favorite indicator—isn’t a measurement
• Lee’s Zeroth Law of Measurement
5454
Thanks
Shamelesscommercial
message