Using Performance Models to Design Self …menasce/cs672/slides/autonomic...1 1 Using Performance...

Using Performance Models to Design Self-Configuring and Self-Otimizing

Computer Systems

Prof. Daniel Menascé Department of Computer Science

E-Center for E-Business George Mason University

Fairfax, VA, USA Menasce@cs.gmu.edu

www.cs.gmu.edu/faculty/menasce.html

Huge number of devicesHuge number of data sourcesMany different data formats

Heterogeneous devicesWidely varying capacity

Wired and wirelessWidely varying QoS requirements

Node failuresConnectivity failures

Security attacksLimited battery power

Characteristics of the new generation of distributed software systems

qHighly distributedqComponent-based (for reusability)qService-oriented architectures (SOA)qUnattended operationqHostile environmentsqComposed of a large number of “replaceable”

components discovered at run-time qRun on a multitude of (unknown and heterogeneous)

hardware and network platforms

Requirements of Next Generation of Large Distributed SystemsqAdaptable and self-configurable to changes in

workload intensity:Ø QoS requirements at the application and

component level must be met.

qAdaptable and self-configurable to withstand attacks and failures:Ø Availability and security requirements must be

self-configurable, self-optimizing, self-healing,and self-protecting

Important Technologies

q Web Services:§ SOAP, UDDI, WSDL

q Grid Computingq Peer to Peer Networksq Wireless Networkingq Sensor and ad-hoc networks

Challenges

q Dynamically changing application structure. q Hard to characterize the workload.Ø unpredictableØ dynamically changing servicesØ application adaptation

q Difficult to build performance models.Ø moving target

q Multitude of QoS metrics at various levels of a distributed architectureØ response time, jitter, throughput, availability, survivability, recovery

time after attack/failure, call drop rate, access failure rate, packet delay, packet drop rate.

q Tradeoffs between QoS metrics (response time vs. availability, response time vs. security)

Challenges (cont’d)

qNeed to perform transient, critical time (e.g., terrorism attack or catastrophic failures) analysis of QoS compliance. Steady-state analysis is not enough.qMapping of global SLAs to local SLAsØCost and pricing issues.

qQoS monitoring, negotiation, and enforcement.qPlatform-neutral representation of QoS goals and

contracts.qResource management: resource reservation,

resource allocation, admission control.Ø non-dedicated resources

What we need …

q Design self-regulating (autonomic systems)q Embed within each system component:Ø Monitoring capabilitiesØ Negotiation capabilities (requires predictive modeling power)Ø Self-protection and recovery capabilities (for attacks, failures, and

overloads)q Push more of the desired system features to individual

componentsq Design QoS-aware middlewareØ QoS negotiation protocolsØ Mapping of global to local SLAsØ QoS monitoring

Rest of this talk …

q Novel uses for performance modelsq Two examples of self-regulating systems:Ø A three-tiered e-commerce systemØ QoS-aware software components

q Concluding Remarks

What are performance models good for?

q At the design stage:ØCompare competing design alternatives.§ A large number of low capacity servers vs. a small

number of large capacity servers?

LoadBalancer

WebServers

LoadBalancer.

Site Design

qDuring production:ØMedium and long-term (weeks and months):§ Capacity planning.

Capacity PlanningResponseTime (sec)

Load (sessions/sec)

3 mo. 6 mo. 9 mo.

QoS Goal

qDuring production:ØMedium and long-term (weeks and months):§ Capacity planning.

Ø Short-term (minutes):§ Dynamic reconfiguration.

Automatic QoS Control: Motivation

qModern computer systems are complex and composed of multiple tiers.

Multi-tier Architecture

qModern computer systems are complex and composed of multiple tiers.qThe workload presents short-term variations

with high peak-to-average ratios.

3600 sec

60 sec

Multi-scale time workload variation

q Modern computer systems are complex and composed of multiple tiers.qThe workload presents short-term variations with high

peak-to-average ratios.qMany software and hardware parameters influence the

performance of e-commerce sites.Manual reconfiguration is not an option!Need self-managing systems.

Computer System

ServiceDemand

Computation

WorkloadAnalyzer

PerformanceModelSolver

QoSControllerAlgorithm

arrivingrequests

completingrequests

QoS Controller

(3) (4)

(6) (7)

(10) (11)

QoSgoals

Computer System

ServiceDemand

Computation

WorkloadAnalyzer

arrivingrequests

completingrequests

QoS Controller

(3) (4)

(6) (7)

(10) (11)

QoSgoals

Computer System

ServiceDemand

Computation

WorkloadAnalyzer

arrivingrequests

completingrequests

QoS Controller

(3) (4)

(6) (7)

(10) (11)

QoSgoals

0/ XUD ii =

Computer System

ServiceDemand

Computation

WorkloadAnalyzer

arrivingrequests

completingrequests

QoS Controller

(3) (4)

(6) (7)

(10) (11)

QoSgoals

Computer System

ServiceDemand

Computation

WorkloadAnalyzer

arrivingrequests

completingrequests

QoS Controller

(3) (4)

(6) (7)

(10) (11)

QoSgoals

Controller Interval

measurements fromservers in e-commerce

reconfigurationcommands

controller algorithm

i-th interval (i+1)-th interval

Combined QoS Metric

XXPPRR QoSwQoSwQoSwQoS ∆×+∆×+∆×=

Rw Pw Xw, , and are relative weights that indicate the relativeimportance of response time, throughput, and probability of rejection.

QoS Metric

PQoS∆

RQoS∆ , , and are relative deviations of theresponse time, throughput, and probability of rejection metrics withrespect to their desired levels.

XQoS∆

QoS Metric

PQoS∆

XQoS∆

The QoS metric is a dimensionless number in the interval [-1, 1].

QoS Metric

PQoS∆

XQoS∆

The QoS metric is a dimensionless number in the interval [-1, 1].

If all metrics meet or exceed their QoS targets, QoS = 0.

Response Time Deviation

),max( max

measured

measuredR RR

−=∆

• = 0 if the response time meets its target.• > 0 if the response time exceeds its target.• < 0 if the response time does not meet its target.

1/)(1 max1

<−≤∆ ∑=

RDQoSK

Rmeasured QoSRR ∆≤−−<− )/1(1 max

Probability of Rejection Deviation

• = 0 if the probability of rejection meets its target.• > 0 and = 1 if the probability of rejection exceeds its target.• < 0 and = -1 if the probability of rejection does not meet its target.

),max( max

measured

measuredP PP

PPQoS −=∆

Throughput Deviation

•• = 0 if the throughput meets its target.• > 0 and < 1 if the throughput exceeds its target.• < 0 and > -1 if the throughput does not meet its target.

),max( *min

QoSmeasured

measuredX

−=∆

),min( min*min XX λ=

Heuristic Optimization Approach

Server Parameter 1

ServerParameter 2

• The space of configurationpoints is searched usinga combinatorial search technique.

• Each point has a QoS valuecomputed through ananalytic performancemodel.

),...,,,( 21 mcccWfQoSr

Server Parameter 1

ServerParameter 2

• The space of configurationpoints is searched usingcombinatorial search techniques.

Currentconfiguration

Server Parameter 1

ServerParameter 2

• The space of configurationpoints is searched usingcombinatorial search techniques.

Newconfiguration

Hill-Climbing Search

1115 1812

16 22 17 18 25 21 19 20

40 28 31 27 29 26 39 32

Beam Search

A Queuing Model is Used to Compute QoS Values

Prototype Configuration

Workstation

100 Mbps Hub

WebServer

ApplicationServer

DatabaseServer

QoSController

WorkloadGenerator

TPC-Wsite

Experiment Results

Arrival rate

0 5 10 15 20 25 30 35

Time (Controller Intervals)

Results of QoS Controller

Arrival Rate (req/sec)

Controlled QoS Uncontrolled QoS

Experiment Results

Arrival rate

0 5 10 15 20 25 30 35

Time (Controller Intervals)

QoS is not met!

Variable inter-arrival and service times of requests

qReal workloads exhibit high variability in:ØTraffic intensityØService demands at various system resources

qNeed to investigate the efficiency of the proposed self-managing technique under these conditionsqConsider variability in requests inter-arrival

time and requests service times at physical resources (e.g., CPU, disk)

Effect of Varying the COV of the Service Time (Ca = 1)

Ca = 1, Cs = 2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Controller Interval

No Controller Beam Search Hill Climbing

Ca = 1, Cs =4

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Controller Interval

Ca = 1, Cs = 1

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Controller Interval

Cs = 1 Cs = 2

Cs = 4

Effect of Varying the COV of the InterarrivalTime (Cs = 1)

Ca = 1, Cs = 1

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Controller Interval

Ca = 2, Cs = 1

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Controller Interval

Ca = 4, Cs = 1

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Controller Interval

Ca = 1 Ca = 2

Ca = 4

Extreme values for Ca and CsCa = 4, Cs = 4

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Controller Interval

Dynamic Controller Interval and Workload Forecasting

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Monitoring Interval

No Forecasting +Dynamic Controller IntervalForecasting + Dynamic Controller Interval

Sensitivity of Controller to SLAs

qNeed to investigate the controller behavior in the case of a variation in the SLAsqWe ran experiments for stricter and more

relaxed SLAsØBase: Rmax= 1.2 sec, Xmin = 5 req/sec, Pmax= 0.05

ØStrict: Rmax= 1.0 sec, Xmin = 7 req/sec, Pmax= 0.03

ØRelaxed: Rmax = 1.5 sec, Xmin = 4 req/sec, Pmax = 0.10

qUsed Ca = Cs = 2

Sensitivity of Controller to SLAs

Controller Sensitivity

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Controller Interval

No Controller Relaxed SLA No Controller Strict SLABeam Search Relaxed SLA Beam Search Strict SLA

Q-Applications and Q-components

Q-component

Q-application

registration

Servicedirectory

Q-component

Q-application

discovery

Servicedirectory

Q-component

Q-application

QoS Negotiation

Q-componentQ-component

Q-component

Q-application

Service Access

QoS-Aware Software Components: Q-Components

q Engage in QoS Negotiations (accept, reject, counter-offer)q Provide QoS guarantees for multiple concurrent

servicesq Maintain a table of QoS commitmentsq Service dispatching based on accepted QoS

commitmentsq Q-components are the building blocks of QoS-aware

applications

Service 1Service 1

ServiceDispatcher

Service kService k

Architecture of a typical software component

ServiceRegistration

Service 1Service 1

QoSRequestHandler

ServiceDispatcher

Service kService k

Architecture of a Q-component (QoS Negotiation)

ServiceRegistration

Service 1Service 1

QoSRequestHandler

QoSEvaluator

ServiceDispatcher

Service kService k

Table ofQoS commitmentsService

Registration

QoSNegoti-

Service 1Service 1

QoSRequestHandler

QoSEvaluator

ServiceDispatcher

Service kService k

Registration

QoSNegoti-

Service 1Service 1

ServiceDispatcher

Service kService k

PerformanceMonitor

Table ofQoS commitments

Architecture of a Q-component –Service Requests

ServiceRegistration

Service 1Service 1

QoSRequestHandler

QoSEvaluator

ServiceDispatcher

Service kService k

PerformanceMonitor

Registration

QoSNegoti-

Architecture of a Q-component

Successful QoS Negotiation

Client Q-componentQoSRequest(rid,Sid,N,Rmax,Xmin)

Accept(rid,token) Request in ToC

ServiceReq (…,token)

ReplyReq (…)

ServiceReq (…,token)

ReplyReq (…)

EndSession (token)Request removed from ToC

On-time Accepted Counteroffer

CounterOffer (rid,N’,token) Request in ToC

AcceptCounterOffer (token)timeout

Expired Accepted Counteroffer

AcceptCounterOffer (token)

timeout

ExpiredCounterOffer (rid)

Request removed from ToC

Rejected Counteroffer

RejectCounterOffer (token)timeout

Request removed from ToC

Rejected QoS Negotiation

RejectQoSRequest (rid)

Decision Tablefor QoSNegotiation

Accept

Reason Remedy Current Others Decision

OK OK Counter OfferOK Not OK Reject

Not OK & MAXR is violated

OK Reject

Not OK & MAXR is violated Not OK

Reject

MINX & MAXR are

violatedReject

Reason Remedy Current Others DecisionOK OK Counter OfferOK Not OK Reject

MINX & MAXR are

violatedReject

Reason Remedy Current Others DecisionOK OK Counter Offer

OKNot OK: N=1 but others still

violatedReject

Any Decrease N

4. Only Other Requests are Violated

Not OK: MINX

violated or N=0

OK or not OK Reject

OK OK Counter OfferNot OK: MINX is

violated or N=0

OK Reject

Only MINX is violated

Decreasing N reduces X and increasing N increases R. So, there is no solution.

Increase N

Decrease N

X could be increased by increasing N. But this would further violate the QoS of

other classes.

Decreasing N reduces X and increasing N increases R. So, there is no solution.

Not OK: MINX is

violated or N=0

OK or not OK

1. Current and other requests are satisfied

Reject

Only MINX is violated

2. Only Current is Violated

Only MAXR is violated

Decrease N

3. Current Request and Others are Violated

Only MAXR is violated Reject

Concurrency level

Resp. Time

Throughput

current

others

Building a Performance ModelNew Request: Sid = 3, N = 12

Base Matrix of Service Demands (in msec): Table of Commitments (ToC):Commitment

ID Service ID N …1 2 3 1 2 10 …

CPU 25 34 20 2 3 15 …Disk 1 30 50 24 3 1 8 …Disk 2 28 42 31 4 1 20 …

5 2 13 …

Matrix of Service Demands (in msec)

1 2 3 4 5 6CPU 34 20 25 25 34 20Disk 1 50 24 30 30 50 24Disk 2 42 31 28 28 42 31

Vector N: 10 15 8 20 13 12

Service

Building a Performance ModelNew Request: Sid = 3, N = 12

Base Matrix of Service Demands (in msec): Table of Commitments (ToC):Commitment

ID Service ID N …1 2 3 1 2 10 …

CPU 25 34 20 2 3 15 …Disk 1 30 50 24 3 1 8 …Disk 2 28 42 31 4 1 20 …

5 2 13 …

Matrix of Service Demands (in msec)

1 2 3 4 5 6CPU 34 20 25 25 34 20Disk 1 50 24 30 30 50 24Disk 2 42 31 28 28 42 31

Vector N: 10 15 8 20 13 12

Service

Service 0Results:

Service 1Results:

Service 2Results:

Svc No.

No. Dropped Sessions

No. of Sessions % Drop

% Resp. Time Reduction

0 24 440 5 111 19 470 4 92 59 590 10 7

Total 102 1500 7 9

Svc No.

0 52 440 12 211 66 470 14 162 148 590 25 12

Total 266 1500 18 16

Svc No.

0 92 440 21 281 140 470 30 262 263 590 45 20

Total 102 1500 33 24

f = 0.0

f = 0.10

f = 0.25

Real -time Stock Quote Response Time

4 13 18 21 31 35 42 50 55 61 106

111 114

126 133

144 148

sessionID

R_NonQoS R_QoS

Stock Quote Service

Delayed Stock Quote Response Time

1 5 8 15 24 28 34 38 46 49 57 62 109

sessionID

R_NonQoS R_QoS

Stock Quote Service

Concluding Remarks

q Performance models can be used to build QoS controllers for complex multi-tiered systems:ØControlled system provides better QoS values even in case

of high variability in request’s inter-arrival and service timesØShort term workload forecasting improves the QoS,

especially when the workload intensity gets close to system saturation levelØDynamic adjustment of the controller interval length

improves the QoS furtherØEven when basic model assumptions are violated, the

models are robust enough to track the evolution of the performance metrics as the workload and configuration parameters change.

Concluding Remarks (Cont’d)

q Performance models can be used by software components to make admission control decisions.ØQoS components should be able to negotiate QoS

requests and perform admission control ØQoS negotiation overhead is small (it did not

exceed 10% of the CPU service demand in our experiments).

Bibliography

q “On the Use of Online Analytic Performance Models in Self-Managing and Self-Organizing Computer Systems,” D.A. Menascé, M. Bennanni and H. Ruan, in the book Self-Star Properties in Complex Information Systems, O. Babaoglu, M. Jelasity, A. Montresor, C. Fetzer , S. Leonardi, A. van Moorsel, and M. van Steen, eds., Lecture Notes in Computer Science, Vol. 3460, Springer Verlag, 2005.

q “Assessing the Robustness of Self-Managing Computer Systems under Highly Variable Workloads,” M. Bennani and D. Menascé, Proc. International Conf. Autonomic Computing (ICAC-04), New York, NY, May 17-18, 2004.

q “A Framework for QoS-Aware Software Components,” D. Menascé, H. Ruan ,and H. Gomaa, Proc. 2004 ACM Workshop on Software and Performance (WOSP’04), San Francisco, CA, January 14, 2004.

q “On the Use of Performance Models to Design Self-Managing Systems,” D. Menascé and M. Bennani, Proc. 2003 Computer Measurement Group Conference, Dallas, TX, Dec. 71-2, 2003.

q “Automatic QoS Control,” IEEE Internet Computing, January/February 2003, Vol. 7, No. 1.

q “Preserving QoS of E-commerce Sites Through Self-Tuning: A Performance Model Approach,” D. A. Menascé, R. Dodge and D. Barbara, Proc. 2001 ACM Conference on E-commerce, Tampa, FL, October 14-17, 2001.

Using Performance Models to Design Self …menasce/cs672/slides/autonomic...1 1 Using Performance...

Documents

A Data Center by Ulrike Talbiersky, Holger Wichert, Christian Lohrengel, André Augustyniak Case Study Source: D. Menasce, V.A. Almeida, L.W. Dowdy Performance

Prediction-Based Admission Control for IaaS …menasce/papers/CloudCom-Published.pdf · Prediction-Based Admission Control for IaaS Clouds with ... fubica@dsc.ufcg ... based admission

Experimental,Analysis,on,Autonomic, Strategies,for,Cloud ...menasce/cs788/slides/Autonomic-Strategies-for-Cloud.pdfExperimental,Analysis,on,Autonomic, Strategies,for,Cloud,Elas8city,

CS672: Approximation Algorithms Spring 14 Introduction to ...shaddin/cs672sp14/slides/LP1.pdfIntroduction to Linear Programming I Instructor: Shaddin Dughmi. Outline 1 Linear Programming

Agents and Service-Oriented Computing for Autonomic …menasce/cs895/slides/Fall2010/CS895-FaisalSibai-AgentsAndService...Agents and Service-Oriented Computing for Autonomic Computing

Performance Engineering Methodologymenasce/cs672/slides/cs672... · Performance Engineering Methodology ... performance requirements of sub -second response time when a natural disaster

CS672: Approximation Algorithms Spring 2020 Linear ...shaddin/cs672sp20/slides/LP.pdf · Linear Programming Basics 6/37. Basic Facts about LPs and Polyhedrons Fact Feasible regions

February 2000CHEP2000 - Dario Menasce, I.N.F.N. Milano 1

Design of Experiments: One Factor and Randomized …menasce/cs700/files/OneFactorExpDesign.pdf · Factor and Randomized Block Experiments ... want to measure. • Factor: ... ANOVA

A MODELING STRATEGY FOR CLOUD INFRASTRUCTURE … · activity for improving service quality, infrastructure planning, and for tuning system components (Jain, 1991; Menasce and Almeida,

Invited: Performance Evaluation of Heterogeneous Multi ...menasce/papers/403_Menasce.pdfnumber of failures and wasted time due to job restarts. Section VII describes the replication

UBL: unsupervised behavior learning for predicting …menasce/cs788/slides/wicke-d-UBL.pdf · UBL: unsupervised behavior learning for predicting performance anomalies in virtualized

Self-Adaptive Software: Landscape and Research …menasce/cs895/slides/Fall2010/CS895-FaisalSib... · Introduction • Why we need self-adaptive software? o Increasing cost of handling

BY Juliana Menasce Horowitz, Anna Brown and Kiana Cox · 2019. 5. 3. · FOR RELEASE APRIL 9, 2019 BY Juliana Menasce Horowitz, Anna Brown and Kiana Cox FOR MEDIA OR OTHER INQUIRIES:

Experimental Analysis on Autonomic Strategies for Cloud ...menasce/cs788/papers/ICCAC2015-Dupont.pdf · tonomic cloud elasticity model relies on dynamic selection of elasticity tactics

Software, Performance, or Engineering?menasce/papers/wosp02-keynote.pdf · 2002-07-22 · Software, Performance, or Engineering? ... time taken by a Web search engine to return a

Modeling the Tradeoffs Between System Performance and CPU ...menasce/papers/CMG15-355_Menasce.pdf · Modeling the Tradeoffs Between System Performance and CPU Power Consumption Daniel

Geant4 Tracking Test (D. Lunesu)1 Daniela Lunesu, Stefano Magni Dario Menasce INFN Milano GEANT4 TRACING TESTs

The BTeV software infrastructure: a personal view The BTeV software infrastructure a personal point of view Dario Menasce

Paper Presentation Optimizing Failure Prediction to ...menasce/cs788/slides/cs788... · Advanced Learning and Research Institute (ALaRI) Faculty of Informatics, Università della