Boetticher Presentation Promise 2008v2

Using Correlation and Accuracy for Identifying Good Estimators

http://nas.cl.uh.edu/boetticher/publications.html The 4th International Predictor Models in Software Engineering (PROMISE) Workshop

Gary D. Boetticher Nazim Lokhandwala Univ. of Houston - Clear Lake, Houston, TX, USA

[email protected] [email protected]

6362

61

http://nas.cl.uh.edu/boetticher/publications.html


mailto:[email protected]

mailto:[email protected]

http://nas.cl.uh.edu/boetticher/publications.html The 3rd International Predictor Models in Software Engineering (PROMISE) Workshop

Research vs. Reality according to JörgensenTSE ’07: 300+ software est. papers,

76 journals, 15+ Years

-89 89-99 00-04 Total

Algorithm 48 137 70 255

ML 1 32 41 74

Human 3 22 21 46

Misc. 7 19 26 52

68% Algorithm

20% ML12% Human

Paper HumanHihn 91 89%Heemstra 91 62%Paynter 96 86%Jørgensen 97 84%Hill 00 100%Kitchenham 02 72%

JSS ’04: Compendium of expert estimation studies

82% Human

18% Formal

Statement of Problem


((Log (TechGradCourses + (TechGradCourses ^ ((Log TotWShops)/(Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (Log (Log (Log SWProjEstExp))))))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (((ProcIndExp + (Log (Sin MgmtGradCourses)))/(Sin SWPMExp)) + (Sin ((Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Sin SWPMExp)))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Cos (TechGradCourses ^ ((Log SWProjEstExp) / (((Log (ProcIndExp + (Log (TechGradCourses ^ ((Log SWProjEstExp) / (Log SWProjEstExp)))))) - 3) / (ProcIndExp + (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos ((((Log SWProjEstExp) / ((ProcIndExp + (Log (TechGradCourses ^ (TechGradCourses ^ (Log SWProjEstExp))))) / (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (Log (Log (Log SWProjEstExp)))))))))))))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / (TechGradCourses ^ (Log SWProjEstExp))))))))))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))) + ((Log SWProjEstExp) / (Log SWProjEstExp)))))) / (Log (Log (Log (TechGradCourses + (Cos (Log (Log (TechGradCourses ^ (Cos (((((Log SWProjEstExp) / (TechGradCourses ^ (Log SWProjEstExp))) / ((ProcIndExp + (Log (Sin MgmtGradCourses))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp))))))))))))))))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Log ((((Log TotLangExp) / (Log SWProjEstExp)) / (Log SWProjEstExp)) / (Sin SWPMExp))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))))))) + (((((ProcIndExp + (Log (TechGradCourses ^ (Log (TechGradCourses + ((TechGradCourses ^ (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos ((((Log SWProjEstExp) / ((ProcIndExp + (Log (TechGradCourses ^ (Log (TechGradCourses + (Cos (Log (Log (TechGradCourses ^ (Cos (((((Log SWProjEstExp) / (TechGradCourses ^ (Log SWProjEstExp))) / ((ProcIndExp + (Log (Sin MgmtGradCourses))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / (TechGradCourses ^ (Log SWProjEstExp))))))) / (Sin SWPMExp))))))) / (TechGradCourses ^ (Log SWProjEstExp))) / (TechGradCourses ^ (Log SWProjEstExp))) / (TechGradCourses ^ (Log SWProjEstExp))) / (Sin SWPMExp)))

Some Background

2006

http://www.starwarscrawl.com/?id=232

Statement of Problem

How to build human-based estimation models that are accurate, intuitive, and

easy to understand?


TechUGCourses < 45.5| Hardware Proj Mgmt Exp < 6| | No Of Hardware Proj Estimated < 4.5| | | No Of Hardware Proj Estimated < 3| | | | TechUGCourses < 23| | | | | Hardware Proj Mgmt Exp < 0.75| | | | | | TechUGCourses < 18| | | | | | | Hardware Proj Mgmt Exp < 0.13| | | | | | | | TechUGCourses < 0.5| | | | | | | | | TechUGCourses < -1 : F (1/0)| | | | | | | | | TechUGCourses >= -1| | | | | | | | | | Degree < 3.5 : A (4/0)| | | | | | | | | | Degree >= 3.5 : A (5/2)| | | | | | | | TechUGCourses >= 0.5| | | | | | | | | TechUGCourses < 5.5| | | | | | | | | | Degree < 3.5 : F (5/0)| | | | | | | | | | Degree >= 3.5| | | | | | | | | | | TechUGCrses < 2 : A (1/0)| | | | | | | | | | | TechUGCrses >= 2 : F (1/0)| | | | | | | | | TechUGCrses >= 5.5| | | | | | | | | | Degree < 3.5| | | | | | | | | | | TechUGCrs < 10.5 : A (3/0)| | | | | | | | | | | TechUGCrses >= 10.5| | | | | | | | | | | | TechUGCrs<12.5 : F (3/0)| | | | | | | | | | | | TechUGCrses >= 12.5| | | | | | | | | | | | | TechUGCrs<16: A (2/0)| | | | | | | | | | | | | TechUGCrs>15 : A (2/1)| | | | | | | | | | Degree >= 3.5 : F (1/0)| | | | | | | HardProjMgmt Exp >= 0.13 : A (2/0)| | | | | | TechUGCourses >= 18 : A (2/0)| | | | | Hard Proj Mgmt Exp >= 0.75 : F (1/0)| | | | TechUGCourses >= 23 : F (5/0)| | | No Of Hardware Proj Est >= 3 : F (1/0)| | No Of Hardware Proj Est >= 4.5 : A (5/0)| Hardware Proj Mgmt Exp >= 6 : F (4/0)TechUGCrses >= 45.5 : A (2/0)

Some Background

2007

The 4th International Predictor Models in Software Engineering (PROMISE) Workshop


PROMISE 2008 versus 2007

• Sample set: 178 Samples

• One learner Accuracy and Intuitive Results

• Attribute reduction Analysis.

• Relatively Simple models.

The Approach


• Personal Demographics• Age, Gender, Nationality, etc.

• Academic• Courses Undergrad/Grad:

CS, HW, SE, Proj. Mgmt, MIS• Workshops/Conferences:

CS, HW, SE, Proj. Mgmt, MIS• Work

• Programming: Ada, ASP, Assembly, C, C++, COBOL, DBMS, FORTRAN, Java, PASCAL, Perl, PHP, SAP, TCL, VB, Other• Work Experience (HW/SW)• Project Management Exp. (HW/SW)• # Projects Estimated (HW/SW)• Average Project Size

• Domain Experience• Procurement Industry Experience

Estimate 28 Components

Scale Factor

And

Correlation

ApplyMachineLearners

Buyer Admin

Buyer1

Buyern

...

Buyer Software

DistributionServer

Supplier1

Supplier2

Suppliern

:

SupplierSoftware

How user compares to other respondents

Feedback to Users


User’s Estimates

Actual Estimates

Experiments: Data


-5

0

5

10

15

20

25

30

-0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0Correlation

Scale

0

0

0

1

1

1

-0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0Correlation

Scale

-5.0

0.0

5.0

10.0

15.0

20.0

25.0

30.0

0.0 0.2 0.4 0.6 0.8 1.0

Correlation

Scale

0

0.5

1

1.5

2

2.5

-0.4 -0.2 0.0 0.2 0.4 0.6 0.8 1.0Correlation

Scale

Original Data set Experiment 1

Experiment 2 Experiment 3

82.8 -29.4

0.008

29X

Experiments: Tools, Configuration


Outliers Removed WEKA Toolset C4.5 (J48) 1000 Trials 10-Fold Cross Validation

Results: Correlation Only


2-Class Problem: 10 Best (A), 10 Worst (F)

1000 Trials,Accuracy = 41.6%

Attribute Reductionusing WRAPPER


Results: Scale Factor Only






Results: Correlation & Scale Factor






Discussion - 1


Best Estimators

Poorest Estimators

Average Correlation 0.4173 0.3686

Average Scale Factor 2.6198 2.7419

How well does the decision tree from the third experiment apply to all the respondents minus outliers?

Discussion - 2


Scope of effort

Amortization of effort

Reuse can skew estimates (esp. Design for Reuse)

Respondent’s estimates = Boetticher’s estimates

Challenges in component effort estimation

Conclusions

Good accuracy rates,

especially after attribute reduction

Correlation + Scale Factor Intuitive Model

Bridges expert and model groups



Thank You !


References

1) Jorgensen, M., “A review of studies on Expert Estimation of Software Development Effort,” Journal of Systems and Software, 2004.

2) Jørgensen, Shepperd, A Systematic Review of Software Development Cost Estimation Studies, IEEE Transactions on Software Engineering, 33, 1, January, 2007, Pp. 33-53.



Business

Boetticher Presentation Promise 2008v2