Queuing Theory and Stochastic Service Systemscfins.au.tsinghua.edu.cn/.../lectures/lect_01.pdf ·...

Queuing Theory and Stochastic Service Systems

Li Xia

Tsinghua University, 2015 Fall

Syllabus • Instructor

– Li Xia 夏俐, FIT 3-618, 62793029, xial@tsinghua.edu.cn

• Text book – Mor Harchol-Balter, Performance Modeling and Design of

Computer Systems—Queueing Theory in Action, Cambridge Press, 2013. (copy is provided)

• Reference books: – D. Gross, J.F. Shortle, J.M. Thompson, and C.M. Harris,

Fundamentals of Queueing Theory, 4th Edition, Hoboken: Wiley, 2008.

– Leonard Kleinrock, Queueing Systems, vol. 1: Theory, John Wiley, 1975.

– Caltech course (Prof. Adam Wierman): http://courses.cms.caltech.edu/cs147/

2 Li Xia, Tsinghua Univ.

Syllabus • Grading

– Homework: 30% (4 assignments, 1 simulation task, in English) plagiary is prohibited

– Midterm: 20%

– Final Project: 40% (the 9th week)

– Course Interaction: 10%

• Lecture notes and assignments are available online (in English)

– http://cfins.au.tsinghua.edu.cn/personalhg/xiali/teaching/course_queues.htm

3 Li Xia, Tsinghua Univ.

What’s your purpose to take this course

• What do you expect to learn from this course?

– Open discussion

• Let’s see some examples in practice

4 Li Xia, Tsinghua Univ.

Telephone switch • A.K. Erlang studied the problem:

– No automatic switch, operator

– How many how many telephone operators were needed to handle a given volume of calls

– Erlang A / Erlang B formula

Li Xia, Tsinghua Univ. 5

Agner K. Erlang

(1878 –1929)

Danish engineer

Erlang (1909) "The Theory of Probabilities and Telephone Conversations" –

which proves that the Poisson distribution applies to random telephone traffic.

Erlang (1917) "Solution of some Problems in the Theory of Probabilities of Significance in

Automatic Telephone Exchanges"

Beijing Subway

Throughput?

Safety?

More lines Increase buffer So what?

6 Li Xia, Tsinghua Univ.

Railway ticket online booking in 2012 Chinese new year

• Crash of ticket booking system

– Large number of tickets for sale (4million)

– Huge visit requests 秒杀? (billion)

– System architecture is not optimal

• Bandwidth of network

• CPU/RAM of computer

• Business logic – Queue + Feedback, greatly reduce the repeating request

• Other factors…

7 Li Xia, Tsinghua Univ.

Modeling and Analysis

• How to solve it?

– Performance analysis and optimization

– Queueing scheme, increase bandwidth…

Internet

Web server Application server

client Database server

IE browser data input interaction display…

passwd verify cookies/other application…

ticket data booking records… 8 Li Xia, Tsinghua Univ.

Applications in daily life

• Supermarket

– How to define the express line (# of items)?

– How to determine the number of checkouts?

– How long customers have to wait at checkouts?

– Behavior of waiting time during peak-hours

• Line at bank counters

– Multiple lines v.s. single line

– Specialist purpose v.s. generalist purpose

– Number of counters?

9 Li Xia, Tsinghua Univ.

Applications in engineering

• Computer/circuit architecture design – 1 fast disk v.s. 2 slow disks?

– Invest on large buffer v.s. fast CPU?

– Scheduling policy to improve performance

• Communication network design – Buffer size design of switch/router

– Data packet scheduling policy in sensor or mobile network

• …

10 Li Xia, Tsinghua Univ.

List of applications areas

• Production system (machine, different products)

• Computer system (cpu, disk, RAM design)

• Communication network (buffer design, link capacity)

• Transportation system (traffic lights control)

• Bank branches operation (counter/type design)

• Airlines scheduling (takeoff/landing arrangement)

• Data center (optimal control, energy saving)

• Call center (optimize the operators, hotlines,…)

• Post office (multi-class, specialization)

• …

11 Li Xia, Tsinghua Univ.

What’s queue? • A general queuing system

Customer arrival

waiting room

Service facility

Customer departure

r22

r11

r12

r23

r31 r21 r32

r13 r33 1

2

3

r20

r10

r30

γ1

γ2

γ3 Queuing

network:

A single

server queue:

12 Li Xia, Tsinghua Univ.

Terminology in queuing theory

• Basic element in queue

– Arrival pattern, service pattern, number of servers, service discipline, system capacity, customer type,..

• Performance metrics

– Average number of customers

– Queue length, average number of queuing customers

– Throughput

– Response time, sojourn time, system time

– Waiting time, queuing time

13 Li Xia, Tsinghua Univ.

Why we need queuing theory?

• Resource constraints

– Why queues appear? How to make them go away?

• Goal of queuing theory

– Predict the performance

– Design the architecture

– Optimize the parameter/policy

• Counter-intuitive

– Randomness is complicated

– Some examples

14 Li Xia, Tsinghua Univ.

Example 1: CPU design

• A simple model of CPU

– Job arrival at rate λ=3/s, Poisson process

– Job mean size is 1/μ, exponential • i.e., service rate is μ=5/s

– FCFS(first come first serve), buffer is infinite

– assume λ < μ, [question]why?

CPU

buffer

Model of a cpu

λ μ

NOTE: modern CPU may have other features, multi-core/PS, etc. 15 Li Xia, Tsinghua Univ.

CPU design, cont.

• If the arrival rate λ doubles, how to upgrade?

– If want to maintain the same delay of jobs, [question] what you choose?

• A. double μ

• B. less than double μ

• C. more than double μ

– Why? Double μ will cut the delay in half

• prove with M/M/1 queuing theory

• Physical intuition, time speeds up with scale 2

16 Li Xia, Tsinghua Univ.

Example 2: Lines in bank

• Customer arrival in Poisson with rate λ

• Counter service rate is μ, exponential

• FCFS, infinite waiting capacity

3

3 lines 1 line

17 Li Xia, Tsinghua Univ.

L1=1.5, L2=0.237

W1=0.5min, W2=0.079min

T1=1min, T2=0.579min

Lines in bank, cont.

• Assume μ = 2/min, λ = 1/min

– queue length,

– waiting time,

– response time,

• [question] how is the following queue?

33

L3 = 0.5

W3 = 0.1667

T3 = 0.3333

18 Li Xia, Tsinghua Univ.

Example 3: many slow v.s. one fast

• CPU selection:

– 1 core CPU with 3GHz freq.

– 3 core CPU with 1GHz freq.

• Which one has a better mean response time?

33

3

19 Li Xia, Tsinghua Univ.

• Depend on the variability of jobs

– Job size variability is high, choose many slow CPU

– Job size variability is low, choose one fast CPU

• Exponential distr.: coefficient of variation(cv) = 1; – For the case of M/M/c and M/M/1, the latter is better

• Uniform distr. or deterministic: cv < 1;

• Hyper-exponential distr. or other distr. (PH, MAP): cv > 1. (self-similarity of Internet traffic)

– If workload is low, one fast is preferred

– If jobs are preemptible (priority, stop, resume)

• One fast is preferred

many slow v.s. one fast, cont.

20 Li Xia, Tsinghua Univ.

many slow v.s. one fast, cont. • many slow v.s. a few fast, widely exist in practice

– Power allocation in data center with server farm

• Fast freq., more power consumption, green data center

– Bandwidth partition in communication systems

• Small chunks of bandwidth, TDMA/FDMA/CDMA …

– Road network in transportation

• Few wide roads v.s. many small roads (Beijing’s problem) – Traffic is bursty with high variability, prefer many slow

– etc. Consider economic factors…

– Service rate control in Jackson network

• (Xia and Shihada, IEEE-TAC 2013) 21 Li Xia, Tsinghua Univ.

Example 4: Closed queueing network

• Model the intensive traffic with N capacity of network

– Batch system, intensive queue with limited capacity, etc.

1

1

3

0.5

0.5

N=6 jobs

1

1

3

22 Li Xia, Tsinghua Univ.

Closed queueing network, cont.

• If we double the speed of server 1

– How it effects the response time of job?

– How it effects the throughput? • [Answer] only change by a small amount

• Suppose N is very large, how is above question?

– Change 0, if N ∞

• What if N is very small

– If N=1, changed amount is large

23 Li Xia, Tsinghua Univ.

Closed queueing network, cont.

• What if the queueing network is open?

– remarkable improvement of throughput and average response time

0.5

0.5

1

1

3

1

1

3

24 Li Xia, Tsinghua Univ.

Example 5: Task assignment in a server farm

• Front-end dispatcher, web server farm, assign task among back-end servers

– used in engineering, Cisco/IBM network device

1

2

Arrivals Dispatcher

(Load Balancer)

25 Li Xia, Tsinghua Univ.

• Task assignment policy

– Determine the task should go to which server

– Based on the system state, policy in MDP

• Different policy

– Random

– Shortest-Queue (SQ)

– Size-Interval-Task Assignment (SITA)

– Least-Work-Left (LWL)

– Central-Queue (CQ)

• Question: which one has best mean response time?

Task assignment, cont.

26 Li Xia, Tsinghua Univ.

• Answer

– Depend on the property of job size • If job size is known, LWL is usually the best

• LWL = CQ ?

– If server discipline is processor-sharing (PS) • SQ is the near optimal

• Task assignment problem

– FCFS/PS, modeled as an MDP optimization problem

– Minimize variance of response time, rather than the mean response time

– Variance (fairness, risk) v.s. Mean (social welfare)

Task assignment, cont.

Chinese Proverb: 不患寡而患不均 27 Li Xia, Tsinghua Univ.

Example 6: Scheduling

• How service disciplines affect response time?

– FCFS, first come first serve

– LCFS, last come first serve

– Random

– [answer] all the same

28 Li Xia, Tsinghua Univ.

Scheduling, cont.

• What if PR-LCFS, preemptive-resumed LCFS?

– Depends on the randomness of job size

• High randomness, big improvement

• No randomness, twice worse

29 Li Xia, Tsinghua Univ.

Summary of examples

• Why counter-intuition?

– Randomness of queuing

– Interactions among customers and servers

• Toy example, but many insights

– Models

– Analysis

– Design

– Optimization

– …

30 Li Xia, Tsinghua Univ.