Aditya P. Mathur Professor, Department of Computer Science, Associate Dean, Graduate Education and...

Aditya P. Mathur

Professor, Department of Computer Science,

Associate Dean, Graduate Education and International

Programs

Purdue University

Monday December 20, 2004. University of Paderborn, Paderborn, Germany.

Software Cybernetics:Progress and Challenges

Cybernetics

[n] the field of science concerned with processes

of communication and control (especially the

comparison of these processes in biological and

artificial systems)

Software Cybernetics

[n] the field of science concerned with processes of communication and control in software systems.

Sample Problems

Control of the software test process:How much and what additional effort is to be applied to achieve the desired quality objective under time/cost constraints?

Optimal selection of tests:What is an optimal set of tests for achieving the desired quality objective given time constraints?

Sample Problems (continued)

Software performance controlHow best to adjust software parameters so that an optimal level of performance is maintained?

Control of the software development process:What is an optimal set of process variables required to achieve delivery objectives within cost/time constraints?

Approaches

1. Use instinct and experience.

2. Use (1) supported by quantitative tools.

(b) Use simulation: “forward” approach.

(c) Use (a) plus feedback control: “inverse” approach.

Software cybernetic approach..

(a) Use simulation: “forward” approach.

Closed Loop (feedback) Control

Specifications

ProgramEffort +

f(e)Additionaleffort What is f ?

-

RequiredQuality

rQ

ObservedQuality

oQ

oQr

Qe −=

Sample Problem Scenario

cp1 cp2 cp3 cp4 cp5 cp6 cp7 cp8 cp9

cpi = check point i

rf

schedule set bythe manager

Approximation of how r is likely to change

r0 observed

deadline

r -

num

ber

ofre

mai

ning

err

ors

t- time

t0

Our Approach

Controllerrerror(t)

’

w’f

+

+

wf+wf

+

wf+wf

+

robserved(t)

rexpected(t)

Actual STP

sc r0

STP State Model

sc r0

Initial Settings(wf,)

wf

Test Manager

wf: workforce : quality of the test process

Decision Support via Feedback

ActualProcess

ParameterEstimation

ProgressMetrics

Process Model

ModelParameters

FeedbackController Estimated Future

Schedule Deviation

Sug

gest

edD

ecis

ion

Cha

nges

Mgmt.Control

ManagementDecisions

-Predicted Schedule

-

+Desired Schedule

Act

ual

Sch

edul

e

+

EstimationError

Challenges (Sources of difficulty)Physical Logical (software)

Laws of physics Yes No

Object controlled Physical Logical (and Humans)

Time dependence Stationary and non-stationary

Non-stationary

Rate of change Very slow to very fast

Slow

Parameter variability Relatively low High

Overcoming the challenge: Understanding the problem

Partnership amongst researchers and practitioners.

Is the problem real?

Are the existing solutions adequate?

Overcoming the challenge: Solving the problem

Parternership amongst researchers and practitioners.

Develop realistic models

Develop parameter estimation methods.

Develop ways to incorporate parameter estimation into the development process.

Details and Case Studies

Razorfish

Guidant Corporation

Sun Microsystems

Physical and Software Systems: An Analogy

Dashpot

Rigid surface

External force

Xequilibrium

X: Position

Number of remainingerrors

Spring Force

Effective Test Effort

Block

Software

Mass of the blockSoftware

complexity

Quality of thetest process

Viscosity

Xcurrent

SpringTo err isHuman

Physical Systems: Laws of Motion [1]

First Law:

Every object in a state of uniform motion tends to remain in that state of motion unless an external force is applied to it.

Does not (seem to) apply to testing because the number of errors does not change when no external effort is applied to the application.

Physical Systems: Laws of Motion [2]

Newton’s Second Law:

The relationship between an object's mass m, its acceleration a, and the applied force F is F = ma.

CDM First Postulate:

The relationship between the complexity Sc of an application, its rate of reduction in the number of remaining errors, and the applied effort E is E=Sc .

r..

Physical Systems: Laws of Motion [3]

Third Law:

For every action force, there is an equal and opposite reaction force.

When an effort is applied to test software, it leads to (mental) fatigue on the tester.

Unable to quantify this relationship.

CDM First Postulate

The magnitude of the rate of decrease of the remaining errorsis directly proportional to the net applied effort and inverselyproportional to the complexity of the program under test.

€

˙ ̇ r = E

sc⇒ E = ˙ ̇ r sc

This is analogous to Newton’s Second Law of motion.

CDM Second Postulate

The magnitude of the effective test effort is proportional to theproduct of the applied work force and the number of remaining errors.

for an appropriate .

Analogy with the spring:

Note: While keeping the effective test effort constant, a reduction in r requires an increase in workforce.

€

eet = ζwf r

CDM Third Postulate

The error reduction resistance is proportional to the errorreduction velocity and inversely proportional to the overallquality of the test phase.

€

er = ξ1

γ˙ r

for an appropriate .

Analogy with the dashpot:

Note: For a given quality of the test phase, a larger error reduction velocity leads to larger resistance.

State Model

€

˙ r

˙ ̇ r

⎡

⎣ ⎢

⎤

⎦ ⎥=

0 1

−ζ wf

sc−

ξ

γ sc

⎡

⎣

⎢ ⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥ ⎥

r

˙ r

⎡

⎣ ⎢

⎤

⎦ ⎥+

0

1

sc

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥Fd

r

˙ r

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥=

1 0

0 1

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

r

˙ r

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

Fd: Disturbance

x(t) = Ax(t) + B u(t).

etr eeE −=Force (effort) balance equation:

Computing the feedback: Question

Question:

What changes to the process parameters will achieve the desired r(T+T) ?

r(T): the number of remaining errors at time T

r(T+T): the desired number of remaining errors attime T+T

Given:

Computing the feedback: Answer

From basic matrix theory:

The largest eigenvalue of a linear system dominates the rate of convergence.

Hence we need to adjust the largest eigenvalue of the system so that the response converges to the desired value within the remaining weeks (T). This can be achieved by maintaining:

teTrtTr −=+ max)()( λObtain the desired eigenvalue.

Computing the feedback-Calculations (λmax)

Compute the desired λmax

teTrtTr −=+ max)()( λGiven the constraint:

We know that the eigenvalues of our model are the roots of its characteristic polynomial of the A matrix.

Computing the feedback-Calculations (λmax)

[ ]

c

f

c

cc

f

s

w

s

ss

wAI

ˆ

ˆ

ˆ

ˆ1

detdet

2 ζλ

γ

ξλ

γ

ξλ

ζλ

λ

++=

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

+

−=−

fwff wwandwhere +=+= ˆˆ

We use the above equation to calculate the space of changes to w and such that the system maintains its desired eigenvalue.

f

A Flow Model of Incremental Software Development/Test

TestSpecs Test

Authoring

Feat.Specs

TestCode

Coding

TestVerification

& Correction Reg.Cases

RegressionTesting

KnownDefects

CodeDebugging

ProjectCode

LatentDefects

Workforce Allocation

Workforce allocated to particular tasks Effort is split across all active tasks

State-Model [Example] Equations

tkPkudtkud ddTT ⋅−=+ )()()(

System State Progress

Feature Coding (fc) Code Debugging (dr) Test Authoring (ta) Test Debugging (td) Regression Testing (rr)

Defect Model Development Testing

tkPkfctkfc fc ⋅+=+ )()()(

tkPkudtkud diDD ⋅+=+ )()()(

cn

bb w

w

rcrcP

⋅+−= )1(

Variable Productivity Equation

Human Productivity Workload Dependent (Csikszentmihalyi,’88)

•rb – Base Work rate

•c – Fractional size-dependent increase

•wc – Current workload size

•wn – Nominal workload size

The “Productivity” Eqns.

[ ]))()(()1()()( kfckfcwrcrcdkwfkP goalfc

fcfcfcfcifcfc −⋅−= +

Process Productivity (E.g. Feature Coding)

Defect Introduction

Defect Detection (Cangussu et al., ’02)

)()()( kPikPikP drdrfcfcdi ⋅+⋅=

[ ])()()()( . kPkPkudckP rrtdTedd +⋅=

Control Strategy

Model Predictive Control

timek k+Pk0

Desired behavior

Actual Behavior

Control Input Prediction Horizon

Predicted FutureBehavior

CalculatedControl Input

2,2,

21

,)(Q

Pkk

Q

Pkkd uxxkJ Pkk

p

++ +−= +

Model Predictive Control Select Cost Functional

E.g. Q1,Q2 := positive definite

Calculate

where S[xk, uk,k+P] xpk,k+P

)(min,

kJPkku +