Carnegie Mellon University Computer Science Department 1 OPEN VERSUS CLOSED: A CAUTIONARY TALE Bianca Schroeder Adam Wierman Mor Harchol-Balter Computer

Carnegie Mellon UniversityCarnegie Mellon UniversityComputer Science Department

1

OPEN VERSUS CLOSED:OPEN VERSUS CLOSED:A CAUTIONARY TALEA CAUTIONARY TALE

Bianca Schroeder Bianca Schroeder Adam WiermanAdam Wierman Mor Harchol-Balter Mor Harchol-Balter

Computer Science DepartmentComputer Science DepartmentCarnegie Mellon UniversityCarnegie Mellon University

To appear at NSDI 2006presenter:吳泰廷


2

standardsystem

newsystem

old

new

new system hassmaller response time!

This comparisonrequires testing the two systems

on realistic workloads

THE RESEARCH PROCESSTHE RESEARCH PROCESS


3

INTRODUTIONINTRODUTION

Need system models that “accurately represent" the real system.

Representing a system accurately involves many things: bottleneck resource behavior, the scheduling of requests at that bottleneck, workload parameters such as the distribution of service request demands…….

One factor that researchers typically pay little attention to is whether the job arrivals obey a closed or an open system model.


4

We show that closed and open system models yield significantlydifferent results, even when both models are run with the same load and service demands.

Conclude with guidelines for choosing a system model.


5

think

receivesend

server

CLOSED SYSTEM MODEL CLOSED SYSTEM MODEL

User requests web page, receives page,reads page, clicks on new link

Closed SystemClosed System

N=MPL(multiprogramming level)


6

1:01.12 ip1 GET a.gif HTTP/1.01:01.20 ip2 GET b.htm HTTP/1.01:01.25 ip1 GET c.jpg HTTP/1.01:01.27 ip1 GET d.txt HTTP/1.01:01.28 ip3 GET a.htm HTTP/1.01:01.35 ip4 GET d.gif HTTP/1.01:01.45 ip2 GET e.htm HTTP/1.0::

Trace driven

OPEN SYSTEM MODELOPEN SYSTEM MODEL

service demands

x x x

server

new arrivals

arrival times

file sizes from trace

next arrivaltime from trace

Open SystemOpen System


7

Distribution driven

Use distributions ofinterarrival times and service demands (typically using trace info)

x x x

server

new arrivals

OPEN SYSTEM MODELOPEN SYSTEM MODEL

interarrival time dist.

service demand dist.

sample dist.

sample dist.

Open SystemOpen System


8

OPEN MODELOPEN MODEL

CLOSED MODELCLOSED MODEL

Arrivals are independentof completions

Arrivals are completelydependent on completions

There is no max number ofsimultaneous users

There is a fixed populationof users, called the

Multi-Programming-Level (MPL)


9


10

WEB WORKLOADWEB WORKLOADGENERATORSGENERATORS

CLOSED MODELCLOSED MODEL

OPEN MODEL OPEN MODEL

SurgeSPECWeb

TPC-WSclientRUBiS

WebBenchWebjamma

1. Workload generators for thesame purpose use differentsystem models!

2. It’s often not clear which model workload generatorsuse!

Do you use an open or closed Do you use an open or closed model?model?


11

NEITHER THE NEITHER THE OPEN OR OPEN OR CLOSEDCLOSED

MODEL IS MODEL IS COMPLETELY COMPLETELY

REALISTICREALISTIC


12

x x xnew arrivals

server

thinksend receive

leave system

with probability qreturn to the system

PARTLY-OPEN MODELPARTLY-OPEN MODEL

PARTLY-OPEN SYSTEMPARTLY-OPEN SYSTEM


13

What is the impact of the choice of an open or

closed model?

OUR GOALOUR GOAL


14

HOW DO WE HOW DO WE COMPARE COMPARE OPEN AND OPEN AND

CLOSED CLOSED SYSTEMS?SYSTEMS?

CLOSEDCLOSED

OPENOPEN

1. Fix the service distribution acrossthe systems

2. Fix the load across the systems

load depends onlyon mean arrival rate and mean service demands

load depends onMPL, think times, mean ofservice demands, variabilityof service demands …

adjust load using thethink time

adjust load using thearrival rate


15

How do open and closedHow do open and closed response times compare? response times compare?

FCFS schedulingopen Poisson arrival processclosed Exponential think times


16

load0 0.25 0.5 0.75 1

mea

n r

esp

on

se

tim

e1000

100

10


Open

Closed (MPL=10)

CLOSED CLOSED <<<< OPENOPEN


17

load0 0.25 0.5 0.75 1

mea

n r

esp

on

se

tim

e1000

100

10

Open

Closed (MPL=10)

Closed (MPL=100)

Closed (MPL=1000)CLOSED CLOSED OPENOPEN



18

OPEN MODELOPEN MODEL CLOSED MODELCLOSED MODELVSVS

CLOSED CLOSED OPEN AS MPL OPEN AS MPL GROWSGROWS

Schatte [36, 37] proves formally that as Ngrows to infinity, a closed FCFS queue converges to anopen queue.


19

low variability high variability

mea

n r

esp

on

se

tim

e

1500

1000

500

Open

Closed (MPL=10)Closed (MPL=100)

Closed (MPL=1000)Web

Workloads

How quickly does Closed How quickly does Closed Open?Open?


20

There principlesThere principles

1.For a given load, mean response times are significantly lower in closed systems than in open systems.

2. As the MPL grows, closed systems become open, but convergence is slow for practical purposes.

3.While variability has a large effect in open systems, the effect is much smaller in closed systems.


21


closed model?

OUR GOAL OUR GOAL

It matters a lot!

1. What is the impacton the effectivenessof scheduling?

2. What is the impactin practice?


22

FCFS (First-Come-First-Served)

PS (Processor-Sharing)

PESJF (Preemptive-Expected-Shortest-Job-First)

SRPT (Shortest-Remaining-Processing-Time-First)

PELJF (Preemptive-Expected-Longest-Job-First)


23

SCHEDULING IS A KEY SCHEDULING IS A KEY COMPONENT OF SYSTEM COMPONENT OF SYSTEM

DESIGNDESIGN

Improved design Shortest Remaining

Processing Time (SRPT)

Standard design Processor Sharing (PS)

WEB SERVERSWEB SERVERS

Does the effectiveness of schedulingDoes the effectiveness of schedulingdepend on the system model (open vs. closed)?depend on the system model (open vs. closed)?

Compare using a workload generator


24

SCHEDULING IN OPEN SCHEDULING IN OPEN SYSTEMSSYSTEMS

OPENm

ean

res

po

ns

e t

ime

1000

600

300

0

load0 .25 .5 .75 1

PLJFFCFSPSSRPT How do the closed How do the closed

results compare?results compare?


25

CONTRASTING THE IMPACT CONTRASTING THE IMPACT OF SCHEDULINGOF SCHEDULING

OPEN CLOSEDm

ean

res

po

ns

e t

ime

1000

600

300

0

load0 .25 .5 .75 10 .25 .5 .75 1

load

PLJFFCFSPSSRPT

PLJFFCFSPSSRPT


26


27

Three priciplesThree priciples1. While open systems benefit significantly from scheduling with respect to response time, closed systems improve much less.

2. Scheduling only significantly improves response time in closed systems under very specific parameter settings: moderate load (think times).

3. Scheduling can limit the effect of variability in both open and closed systems.


28


closed model?

OUR GOALOUR GOAL

It matters a lot!Especially when evaluating

scheduling policies

What is the impactin practice?


29

OPEN VS OPEN VS CLOSEDCLOSED

IN PRACTICEIN PRACTICE

3 CASE STUDIES3 CASE STUDIES

1. Serving static web content

2. Database backend ofan e-commerce site

3. Auctioning web site

testbedimplementation

trace-basedsimulation


30

Case studyCase study Open generatorOpen generator Closed Closed generatorgenerator

Scheduling Scheduling policiespolicies

Static web (LAN)Sclient on

World Cup trace

Modified Sclient on World Cup

tracePS, SRPT

E-commerce Modified TPC-W TPC-W PS, PESJF

Auctioning

Trace-based simulation

(top 10 auction site trace)

Trace-based simulation

(top 10 auction site trace)

PS, SRPT


31

OPEN VS OPEN VS CLOSEDCLOSED

IN PRACTICEIN PRACTICE

OPEN CLOSED

mea

n r

esp

on

se

tim

e

PSSRPT

PS

SRPT

300

200

100

load0 .25 .5 .75 10 .25 .5 .75 1

load

STATIC WEB SERVERSTATIC WEB SERVER

Different models give different conclusion about benefits of SRPT

MPL=50


32

OPEN CLOSED

PS

PS

mea

n r

esp

on

se

tim

e10

8

4

0

load load

PS

SRPT

PS

SRPT

20

14

7

0

load0 .25 .5 .75 10 .25 .5 .75 1

load

PESJFPESJF

E-COMMERCE SITEE-COMMERCE SITE

AUCTION SITEAUCTION SITE

MPL=50

MPL=50


33


closed model?

OUR GOAL OUR GOAL TODAYTODAY

It matters a lot in practice!Especially when evaluating scheduling policies

How can we identify whether to use an open

or closed model?


34

A MORE REALISTIC A MORE REALISTIC ALTERNATIVEALTERNATIVE

x x xnew arrivals

server

thinksend receive

leave system



What parameters affect the load?What parameters affect the load?Does think time affect the load?Does think time affect the load?

How do think times affect response times?How do think times affect response times?


35

FITTING A PARTLY-OPEN FITTING A PARTLY-OPEN MODELMODEL

12 ip1 GET a.gif HTTP/1.020 ip2 GET b.htm HTTP/1.025 ip1 GET c.jpg HTTP/1.027 ip1 GET d.txt HTTP/1.028 ip3 GET a.htm HTTP/1.035 ip4 GET d.gif HTTP/1.045 ip2 GET e.htm HTTP/1.0::

Trace

service demands

file sizes from trace

PARTLY-OPEN PARTLY-OPEN


36

FITTING A PARTLY-OPEN FITTING A PARTLY-OPEN MODELMODEL

12 ip1 GET a.gif HTTP/1.020 ip2 GET b.htm HTTP/1.025 ip1 GET c.jpg HTTP/1.027 ip1 GET d.txt HTTP/1.028 ip3 GET a.htm HTTP/1.035 ip4 GET d.gif HTTP/1.045 ip2 GET e.htm HTTP/1.0::

Trace

PARTLY-OPEN PARTLY-OPEN

Fitting the interarrival times

• Distinguish userse.g. use ip address in a web trace

• Identify user session boundaries Use periods of inactivity of length > timeout

<Ti,fi1,fi2….>


37

CHOOSING A TIMEOUT VALUECHOOSING A TIMEOUT VALUE

Nu

mb

er

of

ses

sio

ns

2e5

1e5

00 30min

Timeout length

financial

world cup

dept store


38

THE EFFECT THE EFFECT OFOF

THINK TIMETHINK TIME


1 10 100 1000

mean think time

300

200

100

0

mea

n r

esp

on

se

tim

e

SRPT

PS


39

CLOSEDCLOSEDOPENOPENq1q0

x x xnew arrivals

server

thinksend receive

leave system



number of requests per visit ↑number of requests per visit ↓ ? ?

A MORE REALISTIC A MORE REALISTIC ALTERNATIVEALTERNATIVE

Workload generators are only Open/Closed!Workload generators are only Open/Closed!


40

THE THE TRANSITION TRANSITION FROM OPEN FROM OPEN

CLOSED CLOSED


300

200

100

00 5 10 15 20

PS open

PS closed

PS

SRPTmea

n r

esp

on

se

tim

e

mean number of requests per visit

OPEN CLOSED


41

THE PARTLY-OPEN THE PARTLY-OPEN SYSTEM IN SYSTEM IN PRACTICEPRACTICE

mean number of requests per visit

mea

n r

esp

on

se

tim

e STATIC WEBSTATIC WEB

PSSRPT

0 5 10 15 20

200

100

0

E-COMMERCE SITEE-COMMERCE SITE

0 5 10 15 20

9

6

3

0

PSPESJF

AUCTIONINGAUCTIONING

0 5 10 15 20

15

10

5

0

PSSRPT


42

PS

SRPT PS SRPT

OPENOPEN CLOSEDCLOSED

PSSRPT

PARTLY-OPENPARTLY-OPEN

VSVS

THESE DIFFERENCES ARE THESE DIFFERENCES ARE IMPORTANT IN PRACTICEIMPORTANT IN PRACTICE


43

Two PrinciplesTwo Principles

1. A partly-open system behaves similarly to an open

system when the expected number of requests per

session is small (≤ 5) and similarly to a closed system

when the expected number of requests per session is

large (≥ 10 as a rule-of-thumb).

2. In a partly-open system, think time has little effect on

mean response time.


44

CHOOSING A SYSTEM MODELCHOOSING A SYSTEM MODELWeb workloads

Open or closed?

Use a partly-open model...

1. Large corporate web 2. CMU web server3. Online department store 4. Science institute (USGS)5. Online gaming site6. Financial service provider 7. Supercomputing web site 8. Kasparov-DeepBlue match 9. Site seeing “slashdot effect” 10. Soccer world cup


45

CHOOSING A SYSTEM MODELCHOOSING A SYSTEM MODELWeb workloads

Open or closed?

Use a partly-open model...

...to decide which is more accurate



46

HOW TO HOW TO CHOOSECHOOSE

A A SYSTEM SYSTEM MODELMODEL

Gathera

trace

How many simult. users are

there?

Fit a partlyopen modelto the trace

OPEN ≈ CLOSED

>>1000

else

What is theexpected num.

of visits?

OPEN CLOSED???

<5 5-10 >10

Me

an

nu

m. o

f vi

sit

s

15

10

5

00 30min

Timeout length

world cup

dept store

financial


47


48


49

CHOOSING A SYSTEM MODELCHOOSING A SYSTEM MODEL<5 expected visits

>10 expected visits

CLOSED

5-10 expected visits

Web WorkloadsOPEN

PARTLYOPEN



50

CHOOSING A SYSTEM MODELCHOOSING A SYSTEM MODEL<5 expected visits

1. Large corporate web 2. CMU web server

4. Science institute (USGS)6. Financial service provider

8. Kasparov-DeepBlue match 9. Site seeing “slashdot effect”

>10 expected visits5. Online gaming site

10. Soccer world cup CLOSED

5-10 expected visits3. Online department store

7. Supercomputing web site

Web WorkloadsOPEN

PARTLYOPEN


51

CONCLUSIONCONCLUSION The differences in behavior of closed, open,and partly-open

systems.

These principles underscore the importance of choosing the appropriate system model.

Our findings provide guidelines for choosing whether an open or closed model is the better approximation based on characteristics of the workload.

Understanding the appropriate system model is essential to understanding the impact of scheduling.

Documents

Carnegie Mellon University Computer Science Department 1 OPEN VERSUS CLOSED: A CAUTIONARY TALE Bianca Schroeder Adam Wierman Mor Harchol-Balter Computer