Performance Evaluation: 30 years in review

Performance Evaluation: 30 years in reviewMartin ReiserGMD – German National Research Center for Information Technology

reiser@gmd.de

MMB’99, 22.–24.September 1999, Trier

Presentation 23. September, 9:00

Very complex systems… where are the limits?

… will it work?

… if so, how well?

… can it work better?

… what will I need tomorrow, in a month, year …?

but why bother anyway?

Host

Switch

Link

ALU

Memory

Channels

30 years?m

any

subs

crib

ers

m tr

unk

line

s

?…

A. K. Erlang, The Post Office Electrical Engineers's Journal 1918

80!

E1,m (ρ) =

ρm

m!ρk

k!k=0m∑

Cutting the Gordian knot

Kleinrock, 1964 Gaver, 1967; Buzen, 1971

Message Independence Central Server

Big bang: BCMP

i

j

closed class s

G(•)

open class r r

i

j

class change pi,r; j ,s

Ns Jobs

Baskett, Chandy, Muntz and Palacios, 1975

LCFS/PRPSIS

E( )

G( )

FCFS

unlimited waiting roomqueue-dependent rates

The PF solution is robust

P(n) C(nii

) Wi,rni ,r

r

station i, class r:

Routing details don’t matter, only mean number of visits does!

Distribution of service-times does not matter, only mean values enter into the solution!

long live the Central Server

The classical models

Delay analysis of data networks (Kleinrock 64)

Time-sharing model (Scherr 67)

Sharing the CPU, i.e. PS (Kleinrock et. al. 64 +)

Central Server (Buzen 71)

Polling, token-ring (Konheim et. al. 74 +)

Aloha, CSMA …(Kleinrock, Lam, Tobagi 73 +)

From trees

3274

3705

3274

370

3705

to meshed networks

From window flow-control

Ti (W) i 1 ni (W 1)

W

Ti (W)ni(W) Ti(W)

to MVA

From Poisson arrivalscall

to MAP…

1

silence talk spurt

2

MPX

20

40

50

60

70

30

Product-Form Qing Theory State-space

Moline (PAP) Erlang MM∞/K

Tandem

Jackson

Gordon/Newell

Convolution

BCMP

Norton's theorem

MM1

Crommelin MDk

MG1LittleCohen

Cyclic Service

Conservation LawsPS

Simon Ando

Courtois

Methods

80

90

MVA

BlockingExotic Qs

Waiting timesG-Networks

customized MVA

PF-Petri Nets

Matrix geom.

Complex tandem

MAP queues

Self-similar trafficLarge deviation th.

UniformizationPerformability

GSPN

Iterative Meth.

DSPNArnoldi

Algebraic meth.

Product-Form Qing Theory State-spaceMethods

Architecture Communication Operation

40

50

60

70

80

90

Capacity planning(BEST1)

Manufacturing

Software Perf. Eng.

Workflow

Netw. mgmt.

Time-sharingVirtual memory

Central Server

Processor arch.Memory hierarchy

(caches)Multiprocessor

Parallel pgm's

Mem. consistencyPrefetching

multithreaded arch.

Telephony

Traffic theory

Delay analysis

Links (Polling)

Flow control

LAN

ATMBroadband

Interconn. netw's

Selfsimilar traffic

Internet, WWW

30

Problems

40

50

60

70

80

90

Capacity planning(BEST1)

Manufacturing

Software Perf. Eng.

Workflow

Netw. mgmt.

Time-sharingVirtual memory

Central Server

Processor arch.Memory hierarchy

(caches)Multiprocessor

Parallel pgm's

Mem. consistencyPrefetching

multithreaded arch.

Telephony

Traffic theory

Delay analysis

Links (Polling)

Flow control

LAN

ATMBroadband

Interconn. netw's

Selfsimilar traffic

Internet, WWW

30

Architecture Communication OperationProblems

PE specialists… work on PE problems cached in PE terms

… build tools

… are integral parts of development teams

… develop (and solve) models of real systems

really?

… use PE methodology in their daily work (planning, managing, tuning)

IS professionals

Success stories

PE-based engineering and development

Network management

Capacity planning

Failures ?

Software performance engineering

PE-basierte F+ESkilled PE specialist group

– is integrated in R&D team– is of sufficient size (5 – 10%)– is highly respected– and contributes to design!

All important design decisions are basedon PE results and analysis

Early prototype supportPE predictions

Case study: IBM Token Ring

VLSI Design

Generally accepted specification process

Simulation data from specification • on several levels of detail

Automatic mask generation

Generation of test patterns

integrated processenvironment

PE in standards

Success story LAN standard IEEE 802

Success story ATM

fix before build

Early attempts in SDLC/HDLC

What is PE?

Is it a science

Is it used by engineers?

Is it used in IS planning/tuning?

Is it a business?

Is it regular part of university curriculum?

…………………………………………………yes

……………………………some

………………little

…………cottage industry

……not really

Quo vadis ?

Stay where we are?

Reach out to main stream?Must make quantum leap in SPE

Must integrate tool into develop-ment processes • robust • Sprache des Benutzers

• accepted in EE, especially communications

• continue existence as niche science

SPE roadmap

Rep.Usage

scenario editor

Workload data

Network capacityplanning

Analytic system structure model

System structure

editor

Capacityplanning

Path-lengthestimation

process

CASE me-thodology

In user’s language

3 grand challenges

Understand self-similar traffic • do we need a new theory? • se we merely shifting loads?

Introduce analytical methods in computerarchitecture (“a gold mine” S. Adve)

to enter into the main stream ofthe CASE process and tools