Measurement and Experiment in Software Enginerring

Measurement and Experimentin Software Enginerring

Bill Curtis

Presented by: Lei Wang

CS5391 Spring 2003

Measurement and Experimentin Software Enginerring – p.1

Abstract

Review the contributions of measurement andexperimentation to the state of the art insoftware engineering.

Discuss the role of measurement indeveloping theoretical models.

Concern for reliability and validity arestressed.


AbstractCurrent approaches to measuring softwarecharacteristics are presented as examples.

Software complexity metrics related toControl flowModule interconnectednessHalstead’s Software Science

Experimental methods in evaluating cause-effectrelationship.

Example programs of experimental research.

Conclusion


Introduction

The core of rigorous scientific proceduresThe development of measurementtechniques.The determination of cause-effectrelationship.

Modeling the process underlying a softwaretask.


Science and MeasurementScientific discipline can be classfied by the degree towhich their analytical approach is theoretical ratherthan correlational.Margenau,1950 and Torgerson, 1958.

The development of scientific theory involves relatingtheoretical constructs to observe data.


Science and MeasurementTHEORY IN A LESS WELL−DEVELOPED DISCIPLINE

DATA

OBSERVABLE

DATA

THEORY IN A WELL−DEVELOPED DISCIPLINE

OBSERVABLE

OPERATIONAL DEFINITION

PRESUMED RELATIONSHIP

EQUATIONCONSTRUCT

MEASURABLE VARIABLE

Figure 1: The structure of theory in science


Science and MeasurementMeasurement quantifies a property of the construct.

Reliablity concerns the extent to which measures areaccurate and repeatable.

Internal consistency

stability

Validity concerns whether a measure represents whatit was designed to access.

Content validity

Predictive validity

Construct validityMeasurement and Experimentin Software Enginerring – p.7

Science and Measurement

SCALEAPPROPRIATE

DESCRIPTION EXAMPLESOPERATIONS

NOMINAL =, 6= CATRGORIESSEX, RACE

JERSEY NUMBERS

ORDINAL <, > RANK ORDERINGSNARDNESS OF MIMERIAL

RANK OF CLASS

INTERVAL +,−EQUIVALENT INTERVALS TEMPERATURE(F ◦ AND C ◦)

BETWEEN NUMBERS CALENDAR TIME

RATING ÷EQUIVALENT INTERVALS TEMPERATURE(K ◦) WEIGHT

AND ASSOLUTE ZERO LINES OF CODE

Table 1: The operations listed fro each scale are appropriate for allscales listed beneath it.


Measurement of softwarecharacteristics

Uses for software metrics

Management information tools

Measures of software quality

Feedback to software personnel

Measures of process and product

They are different.

It will be difficult to develop a metric which canrepresent both.

Mainly deal with measures of product in this paper.


Omnibus approaches toquantifying software

Matric quantify numerous factors underlying concept

Boehm, Brown et al. see Figure 2.

McCall et al.

Higher level constructs in each system represent

The current behavior of the software

The ease of changing the software

the ease of converting or interfacing the system



Device independence

Self-containedness

Accuracy

Completeness

Robustness/integrity

Consistency

Accountability

Device efficiency

Accessibility

Communicativeness

Self-descriptiveness

Structuredness

Conciseness

Legibility

Augmentability

Human engineering

Modifiability

Reliability

Testability

Understandability

Portability

Efficiency

GeneralUtility

As-isUtility

Maintainability

Figure 2: The Boehm et al. software quality modelMeasurement and Experimentin Software Enginerring – p.11


INTEROPERABILI

TY

RELIABLITY

CORRECTNESS

EFFICIENCY

INTEGRITY

USABILITY

MAINTAINABILITY

TESTABILITY

FLEXBILITY

PORTABILITY

REUSABILITY

INTEROPERABILITY

CORRECTNESS

RELIABLIT

YEFFIC

IENCY

INTEGRIT

YUSABILI

TY

MAINTAIN

ABILITY

TESTABILITY

FLEXBILI

TY

PORTABILITY

REUSABILITY

Figure 3: McCall et al.’s tradeoff analysis among software quality factors.Measurement and Experimentin Software Enginerring – p.12

Software Complexity

Computational complexity

Relies on formal mathematical analysis

The quantity aspects of the solutions tocomputational problem

Psychological complexity

The understanding that software development andmaintenance are largely human activities.

Concern the characteristics of software which affectprogrammer performance


Software Complexity

Definition of Complexity:Complexity is a characteristic of the software interfacewhich influences the resources another system willexpend or commit while interacting with the software.

The focus of complexity not merely on the software, buton the software’s interactions with other system.

Explicit criteria are not specified.

Complexity will have different operational definitionsdepending on the criterion under study.


Software Complexity

Steps in modeling an aspect of software complexity:

1. define(and quantify) the criterion the metric will bedeveloped to predict.

2. develop a model of process in the interacting systemwhich will affect this criterion.

3. identify the properties of software which affect theoperation of these process.

4. quantify these software characteristics.

5. validate this model with empitical research.


Control StructureMany metrics having a theoretical base in graph theoryto measure software complexity by accessing thecontrol flow.

Discussing of McCabe’s metric

Define complexity in relation to the decisionstructure of a program.

Complexity metric is the classical graph-theorycyclomatic number indicating the number of regionsin a graph. Demostrated in Figure 4.

The different counting methods developed by Basili,Reiter and Myers.


Control Structure

V(G) = 9

1

2

1

2

3 4

1

2

34

5

6

78

9

V(G) = 2 V(G) = 4

Figure 4: Computation of McCabe’s V (G).Measurement and Experimentin Software Enginerring – p.17

Software ScienceHalstead’s theory of software science

V = (N1 + N2) log2(η1 + η2)

E =η1N2(N1 + N2) log

2(η1 + η2)

2η2

Where V is volume, E is effort. and

η1 number of unique operators

η2 number of unique operands

N1 total frequency of operators

N2 total frequency of operandsMeasurement and Experimentin Software Enginerring – p.18

Software ScienceExperiment evaluation of Halstead and McCabe’smetrics

Both superior to LOC for predicting the time to findand fix error.

Better prediction of outcome may occur when moredisciplined software development practices reducesthe often dramatic performance differences amongprogrammers(see Figure 5).

Both to be valid measures of psychologicalcomplexity.

useful in actual practice.Measurement and Experimentin Software Enginerring – p.19

Software Science

HALSTEAD’S V

200 400 600 800

10

20

30

40

50

DE

VE

LOP

ME

NT

TIM

E (M

INU

TES

)

100

Figure 5: Scatterplot of Halstead’s V against development time fromSheppard et al. Measurement and Experimentin Software Enginerring – p.20

InterconnectednessSeveral metrics developed to access the complexity ofthe interconnectedness among the parts comprising asoftware system

Myers,1976

Model system complexity by developing adependency matrix among pairs of modules baseon whether there is a interface between them

Present two important consideration: Strength of amodule and Coupling between modules.

Yau et al., 1979, Working on validating a generic typemodel.


Experimental Evaluation ofSoftware Characteristics

Cause-Effect Relationships

focus on the evalution of conditional statements andcontrol flow.

Conditional Statement

Nested conditionals – more visible andcomprehensible to programmer.

Branch-to-label conditionals – Obscure the visiblityof embedded conditions.


Conditional StatementNested conditions:IF [condition] THEN [process 1]

OTHERWISE [process 2].

Branch-to-label conditions:IF [condition] GOTO L1

[process 2]

L1 [process 1].

Branch-to-label conditions place a greater cognitiveload on programmers.


Conditional StatementIF-GOTO NESTED BEGIN-END NESTED IF-NOT-END

IF (Condition 1)GOTO L1

IF (Condition 3)GOTO L2

(Process 4) STOPL1 IF(Condition 2)GOTO L3

(Process 2) STOPL2 (Process 3) STOPL3 (Process 1) STOP

IF (Condition 1) THENBEGINIF (Condition 2)

THENBEGIN (Process 1)ENDELSEBEGIN (Process 2)END

ENDELSEBEGINIF (Condition 3)

THEN BEGIN (Process3)ENDELSEBEGIN (Process 4)END

END

IF (Condition 1)IF (Condition 2)

(Process 1)NOT (Condition 2)

(Process 2)END (Condition 2)

NOT (Condition 1)IF (Condition 3)

(Process 3)NOT (Condition 3)

(Process 4)END (Condition 3)

END (Condition 1)

Table 2: Conditional Structures Inverstigated by Sime, Green and Guest.


Conditional StatementIF-GOTO cause more semantic(algorithmic) errors.

Nested language cause primarilysyntactic(grammatical) errors.

Errors were debugged ten times faster in theIF-NOT-END condition.

Sheiderman and Mayer’s Model:A language design should seek to simplify both thedesign and expression of an algorithm.


Conditional StatementTwo types of information: sequence and taxon (Sime,et al., 1977)

Sequence information can easily extracted from nestedlanguage.

Backward tracing is performed much more easily withthe IF-NOT-END construct.

Writing procedure and automated tool reduce thenumber of syntactic errors.

More experiment on Conditional Statement: Arblaster,Sheiderman, Green, etc.


Conditional StatementThe superiority of nested over branch-to-labelconditions.

The advantage of redundant expression of controllingconditions at the entrance to each conditional branch.

The benefits of a software practice may vary with thenature of the task.

A standard procedure for generating the syntax of aconditional statement can improve coding speed andaccuracy.


Control FlowReview only experiments related to structured coding.

The control structures generally allowed understructured coding are show in Figure 6.

It is not clear that structured coding will improve theproductivity of programmers.

Structured coding should reduce the costs ofmaintenance.


Control flow

REPEAT

REPETITION

SEQUENCE

BEGIN

process 1

process 2

END

WHILE CONDITION B0

ENDB0

process

UNTIL CONDTION

process


Control flow

CASE 1:

SELECTION

process 1

IF CONDITION THEN

process 2

process 2ENDIF

ELSE

CASE CONDITION CF

CASE 3:process 3

process 2CASE 2:

ENDCASE

Figure 6: Constructs in structured control flow.Measurement and Experimentin Software Enginerring – p.30

Summary

Measurement and experimentation are complementaryprocess.

Results are far more impressive when they emerge from aprogram of research rather than from one-shot studies.

The rigors of measurement and experimentation requireserious consideration of processes underlying softwarephenomena.

No substitute for sound experimental evidence in arguingthe benefits of a particular software engineering practice orin comparing the relative merits of several practices.


Documents

Measurement and Experiment in Software Enginerring