Upload
cruz-raymond
View
29
Download
2
Embed Size (px)
DESCRIPTION
Investigating Individual Software Development: An Evaluation of the Leap Toolkit. Cam Moore Collaborative Software Development Laboratory Communication & Information Sciences University of Hawaii, Manoa [email protected] http://csdl.ics.hawaii.edu/Research/LEAP/LEAP.html. Outline. - PowerPoint PPT Presentation
Citation preview
(1)
Cam Moore
Collaborative Software Development Laboratory
Communication & Information Sciences
University of Hawaii, Manoa
http://csdl.ics.hawaii.edu/Research/LEAP/LEAP.html
Investigating Individual Software Development: An
Evaluation of the Leap Toolkit
(2)
Outline
Why Is Quality Software Development Important?
Solutions
Thesis
Evaluation
Contributions & Future Directions
(3)
Problem Every Software developer wishes they got home earlier and spent less of their weekend at work.
Software is often:•Late
-Windows 2000•Over budget
-DIA baggage control•Full of defects
-Therac-25
Problem
Solutions
Leap
Evaluation
Contributions
(4)
Solutions
Hope Based (Try Harder)1960s
Product Based (Testing, Review)1970s
Organization Based (CMM, ISO 9000)1980s
Developer Based (PSP, Leap)1990s
Problem
Solutions
Leap
Evaluation
Contributions
(5)
Problems with traditional solutions
Many software process improvement initiatives suffer from one or more of the following problems:
•Heavy weight development constraints
•Organization-level analysis and improvement
•Measurement Dysfunction
•Manual Data Collection
Problem
Solutions
Leap
Evaluation
Contributions
(6)
Supporting Software Developer Improvement
with LEAP LEAP is our design philosophy. All LEAP tools must satisfy four major criteria:
•Light weight
•Empirical
•Anti-Measurement dysfunctional and
•Portable
The Leap toolkit is a reference implementation
Problem
Solutions
Leap
Evaluation
Contributions
(7)
Leap Toolkit Started August 1997 Over 25 public releases
As of 17 Apr. 2000, the size of the Leap Toolkit is:•44KLOC of Java, •2156 Methods, •311 Classes and, •14 Packages
It is available for download from http://csdl.ics.hawaii.edu/Tools/LEAP/LEAP.html
Problem
Solutions
Leap
Evaluation
Contributions
(8)
Leap Toolkit
(9)
General Thesis
LEAP provides a more accurate and effective way for developers to collect and analyze their software engineering data than methods designed for manual enactment.
Problem
Solutions
Leap
Evaluation
Contributions
(10)
Evaluation To evaluate this thesis I broke it into 3 claims:
I. Leap is able to prevent important classes of data error as identified in Disney98.
II. Leap reduces the level of collection stage errors by reducing the overhead of data collection.
III. Leap implements data collection and analyses that enable sophisticated analysis not available in manual methods such as PSP.
Problem
Solutions
Leap
Evaluation
Contributions
(11)
I: Preventing data error I. Leap is able to prevent important classes of data error as identified in Disney98.
Disney’s research discovered the following problems with manual PSP data:
• Students had a 5% error rate in their data
• These errors were significant-Time estimation-Incorrect Yield
Even 95% accuracy is not good enough
Problem
Solutions
Leap
Evaluation
Contributions
(12)
I: Preventing data error The design and implementation of the Leap toolkit address the data error categories:Error Category Leap solution
Calculation error(46%)
Automatedcalculations
Transfer error(20%)
MVCarchitecture
Blank fields(18%)
Default values<Undefined>
Problem
Solutions
Leap
Evaluation
Contributions
(13)
I: Preventing data error Data error categories (cont):
Error Category Leap solution
Entry error (9%) Default valuesPopup menus
Impossible values(6%)
Error checking
Sequence error(1%)
Not an errorProblem
Solutions
Leap
Evaluation
Contributions
(14)
II: Reducing collection error
II. Leap reduces the level of collection stage errors by reducing the overhead of data collection.
Case Study:•16 Students used Leap while developing 8 Java programs in ICS 613
•Data collection:-4 surveys conducted (# responses)-Interviews (11 conducted)-Students’ Leap data (16 data sets)
Problem
Solutions
Leap
Evaluation
Contributions
(15)
II: Reducing collection error
The students were asked to collect three types of data:
•Defects-Students used many different collection methods
•Size-LOCC used for collecting size data
•Time-Two different time collection tools
Problem
Solutions
Leap
Evaluation
Contributions
(16)
II: Reducing collection error
First method of time collection in LeapNaia
Defined Values Default Values Current Time Automatic
Duration
Problem
Solutions
Leap
Evaluation
Contributions
(17)
II: Reducing collection error
Second method of time collection in Leap
Io
Defined Values Current Time One click
operation
AutomaticDuration
Problem
Solutions
Leap
Evaluation
Contributions
(18)
II: Reducing collection error
Using Naia for data entry makes data entry harder
•Students using Naia said data collection was neutral to hard
•Students using Io said data collection was easy to neutral
Problem
Solutions
Leap
Evaluation
Contributions
(19)
II: Reducing collection error
Types of time collection errors•Missing time•Added time•Rounding
Reported use of Naia has high correlation to rounded timesPercentage of times that were a multiple of 5
minutes
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
16 14 7 3 8 1 4 15 2 13 11 10 6 9 12 5
Student
Percentage of times that were a multiple of an hour
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
15 7 14 3 6 8 12 4 13 11 10 1 5 16 2 9
Student
(20)
III: Empirical Investigation III. Leap implements data collection and analyses that enable sophisticated analysis not available in manual methods such as PSP.
Leap allowed me to evaluate 14 different quantitative time estimation techniques.
A
BCActual Size
Planned Size
Avera
ge
Lin
ear
Exp
on
en
tial LOC
Methods
PSP (A, B, or C)
Problem
Solutions
Leap
Evaluation
ContributionsStudent’s
(21)
III: Empirical Investigation I used Leap to calculate the prediction error for each estimation method.•Prediction Error = | estimate - actual |
•Compared methods for each student
•Compared methods for the whole class
Problem
Solutions
Leap
Evaluation
Contributions
actual
(22)
III: Empirical Investigation Individual Students’ Results:
3 of the 16 students had significantly better estimation methods•2 students, their own estimation methods were best (F = 3.93, p < 0.001), (F = 2.25, p = 0.019)
•1 student, exponential regression using actual LOC was best (F = 2.34, p = 0.014)
•5 students, own estimates best
Problem
Solutions
Leap
Evaluation
Contributions
(23)
III: Empirical Investigation Class’ Results:
The students’ own estimates were the most accurate (F = 3.75, p <0.001)
Ignoring the students’ estimates, the Exponential Regression using actual methods was the best (F = 2.40, p = 0.005)
The PSP’s time estimation method was 6th
Problem
Solutions
Leap
Evaluation
Contributions
(24)
Contributions The Leap toolkit•An effective tool for software developer improvement-Reduces overhead-Flexible-Portable
•A novel software engineering tool for:-Education-Research
Problem
Solutions
Leap
Evaluation
Contributions
(25)
Contributions The results of the estimation experiment•Insight into better software estimation techniques
Experience may lead to over-confidenceEstimation Accuracy
-40.00%
-30.00%
-20.00%
-10.00%
0.00%
10.00%
1 2 3 4 5 6
Project
Err
or
% Less Exp
More ExpProblem
Solutions
Leap
Evaluation
Contributions
(26)
Future Directions Replicated studies with more subjects including industrial software developers
Industry adoption of Leap Toolkit
More support for data analysis in Leap Toolkit
Further study into effort estimation
Personal Agents that “observe” the developer
Problem
Solutions
Leap
Evaluation
Contributions
(27)
(28)
Results: Case Study Estimation accuracy unaffected by rounding error•No significant difference between the estimation accuracies of the class (F = 1.71, p = 0.07)
•No significant difference between Naia and Io users (F = 2.70, p = 0.10)
Estimation accuracy
-30.00%
-25.00%
-20.00%
-15.00%
-10.00%
-5.00%
0.00%
5.00%
1 2 3 4 5 6
Project
Err
or
% Naia ave
Io ave
(29)
Results: Case Study There was very little evidence of measurement dysfunction
•Grades not based upon estimation accuracy
•Students did not compare their Leap data
•Most students felt little pressure to make their data look “good”-Most said they felt less pressure-Two with increasing remained very low
(30)
Results: Case Study Students showed traditional software developer improvements
•Increased Productivity
•Improved size estimation skills
•Improved time estimation skills
(31)
Increased Productivity
LOC•EqualSlight improvement
Methods•Equal
Productivity LOC/hr.
0
20
40
60
80
100
120
0 2 4 6 8 10
Project
LO
C/h
r.
Productivity Methods/hr
0.00
2.00
4.00
6.00
8.00
10.00
12.00
0 2 4 6 8 10
Project
Met
ho
ds/
hr
(32)
Productivity Size of methods increased•Projects changed from Command line to GUI
•Last few project were more complicated
Problem
Solutions
Leap
Evaluation
Contributions
LOC/Method
0.00
10.00
20.00
30.00
40.00
50.00
0 2 4 6 8 10
Project
LO
C
(33)
Improved Size Estimation
LOC•Steady Underestimation
Methods•Improvement•BetterDesigns?
Size Estimation Accuracy (LOC)
-100.00%
-50.00%
0.00%
50.00%
100.00%
Project
Est
imat
ion
Err
or
Size Estimation Accuracy (Methods)
-100.00%
-50.00%
0.00%
50.00%
100.00%
150.00%
Project
Est
imat
ion
Err
or
(34)
Improved Time Estimation Class’ time estimation accuracy•Under estimated•Range consistent
Time Estimation Accuracy
-80.00%
-60.00%
-40.00%
-20.00%
0.00%
20.00%
40.00%
60.00%
80.00%
Project
Est
imat
ion
Err
or