Performance Evaluation of Complex Systems: from Large Software

Performance Evaluation of Performance Evaluation of Complex Systems: from Large Complex Systems: from Large Software Architectures to the Software Architectures to the

Cell ProcessorCell Processor

INFN Sezione di [email protected]

http://www.dsi.unive.it/~marzolla

Università di Bologna

Moreno Marzolla

http://www.dsi.unive.it/~marzolla

Moreno Marzolla 2009-05-19, Università di Bologna 2

Talk OutlineTalk Outline

● Motivations● Model-Based Performance Evaluation

Software Architectures Web Services Cell Processor

● Conclusions


MotivationsMotivations

● Large software systems are complex and expensive artifacts

● Their success depends on whether their functional and non-functional requirements are satisfied

● Performance is one of the most important non-functional requirements


How can performances be How can performances be evaluated?evaluated?

● Measurement-based approach Perform direct measures on a running system or

a prototype; use these measurements to identify bottlenecks

● Model-based approach Develope a performance model of the software

system; use the model to mimic the behavior of the system and predict its performances


The “fix-it-later” approachThe “fix-it-later” approach

1.Define the system architecture2.Build and deploy the system3.Evaluate performance through

direct measurements on the running system

4.If performance requirements are not satisfied, go back to step 1


Model-Based System Model-Based System Performance EvaluationPerformance Evaluation

SystemSystemModelModel

PerformancePerformanceModelModel

ResultsResultsandand

FeedbackFeedbackModel Model

EvaluationEvaluation


Performance Evaluation of Performance Evaluation of Software SystemsSoftware Systems


WaterfallWaterfall Software Development Software Development ModelModel

System andSoftware design

RequirementsDefinition

Implementationand unit testing

Integration and system testing

Operation and maintenance

Model-basedPerformance

Evaluation

Measurement-basedPerformance Evaluation







Simulation-Based Software Simulation-Based Software Performance EvaluationPerformance Evaluation

AnnotatedAnnotatedUML ModelUML Model

SimulationSimulationModelModel

PerformancePerformanceResultsResults

Simul. Model Simul. Model EvaluationEvaluation

ArgoUMLargouml.tigris.org

UML SPT Profile

UML-PSIMarzolla and Balsamo, proc. QEST'04

libcppsimMarzolla, proc. ESM'04


The The modelingmodeling cycle cycle

● The UML model is annotated according to the “UML Profile for Schedulability, Performance and Time Specification”

● The annotated UML model is automatically translated into a simulation model Use Case Diagrams ↔ Workloads Activity Diagrams ↔ Simulation Processes Deployment Diagrams ↔ Resources

● The simulation model is executed● Simulation results are inserted back into the

UML model as annotations


ExampleExampleAnnotated Use Case DiagramAnnotated Use Case Diagram

<<ClosedWorkload>>

Request Video

PApopulation = 10PAextDelay = [“assm”, “dist”,

[“exponential”, 1.0/50.0]]


ExampleExampleAnnotated Activity DiagramAnnotated Activity Diagram

Select Service

Handle Selection

Confirm

Initiate Playout

Initialize Player

Send Video

Send Terminate

Terminate Player

PAhost = “ClientWorkstation”PAdemand = [“assm”, “dist”,


PAhost = “ClientWorkstation”PAdemand = [“msrd”, “dist”,

[“constant”, 0.5]]

PAhost = “Internet”PArep = 100PAdemand = [“assm”, “dist”,


PAhost = “VideoServerNode”PAdemand = [“msrd”, “dist”,


PAhost = “VideoServerNode”PAdemand = [“assm”, “dist”,


PAhost = “WebServerNode”PAdemand = [“est”, “dist”,

[“constant, 2.0]]

PAhost = “ClientWorkstation”,PAdemand = [“assm”, “dist”,


PAhost = “ClientWorkstation”PAdemand = [“msrd”, “dist”,



ExampleExampleAnnotated Deployment DiagramAnnotated Deployment Diagram

ClientWorkstation: WebServerNode: VideoServerNode:

Internet:

PArate = 1.0PAschedPolicy = ”PS”

PArate = 2.0PAschedPolicy = ”FIFO”

PArate = 2.0PAschedPolicy = ”FIFO”

Parate = 2.0PAschedPolicy = ”FIFO”


Model development using Model development using ArgoUMLArgoUML


Model analysis using UML-PSIModel analysis using UML-PSI

● UML-PSI (UML Performance SImulator) is a discrete-event simulator written in C++ Command-line tool Parses annotated UML diagrams in XMI format Builds a simulation model Executes the model Performance results (resources utilization,

actions response times, ...) are inserted back into the XMI file as additional tagged values

● http://www.dsi.unive.it/~marzolla/software


Using a different performance Using a different performance modelmodel







QueueingQueueingNetworkNetworkModelModel


QN Model QN Model EvaluationEvaluation

ArgoUMLargouml.tigris.org

UML SPT Profile

Balsamo, Mamprin, Marzolla proc. ESMc'04Balsamo and Marzolla, proc. WOSP'05

QNetworkswww.dsi.unive.it/~marzolla/software/qnetworks


Translating Use Case DiagramsTranslating Use Case Diagrams<<PAopenLoad>>

<<PAclosedLoad>>

λ

PAoccurrence = λ

PApopulation = NPAextDelay = Z

Z

N


Translating Activity diagrams:Translating Activity diagrams:the “easy” casethe “easy” case

A1

A2

A3

R1

R2

R3

R1

R2

R3


Translating Activity diagrams:Translating Activity diagrams:the “not-so-easy” casethe “not-so-easy” case

A1

A2

A3

R1

R2

R1

R2


Evaluation of the QN ModelEvaluation of the QN Model









BoundBoundAnalysisAnalysis

MVAMVA

SimulationSimulation


Evaluation of the QN ModelEvaluation of the QN Model

● QNetworks: a Queueing Networks analysis package for GNU Octave

● Free and Open Source, GPLv3 http://www.dsi.unive.it/~marzolla/software/qnetworks/

● Another good QN tool: Java Modelling Tools http://jmt.sourceforge.net/


The QNetworks packageThe QNetworks package● Single-station queueing systems

M/M/1, M/M/m, M/M/1/k, M/M/m/k, M/M/∞● Bounds on performance

Balanced System Bounds, Asymptotic Bounds, Geometric Bounds

● Product-form QN MVA for single/multiclass QN, closed or mixed networks with

load-dependent service centers Jackson networks Convolution algorithm

● Non product-form QN MVABLO

● Miscellaneous Continuous- and Discrete-time Markov Chains Computation of visit ratios


ReferencesReferences● S. Balsamo and M. Marzolla, Simulation Modeling of UML Software

Architectures, Proceedings of the European Simulation Multiconference, Nottingham - UK, 2003

● S. Balsamo and M. Marzolla, A Simulation-Based Approach to Software Performance Modeling, Proceedings of ESEC/FSE 2003

● M. Marzolla, S. Balsamo, UML-PSI: The UML Performance SImulator, Proc. of the First International Conference on the Quantitative Evaluation of Systems (QEST 2004)

● S. Balsamo, R. Mamprin, M. Marzolla, Performance Evaluation of Software Architectures with Queuing Network models, in Proc. of ESMc'04

● S. Balsamo, M. Marzolla, Performance Evaluation of UML Software Architectures with Multiclass Queueing Network Models, in Proc. Fifth Int. Workshop on Software and Performance (WOSP'05)

● S. Balsamo, M. Marzolla, R. Mirandola, Efficient Performance models in Component-Based Software Engineering, in Proc. 32nd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2007

● http://www.dsi.unive.it/~marzolla/software/qnetworks/


Performance Evaluation of Web Performance Evaluation of Web ServicesServices


Context: the gLite MiddlewareContext: the gLite Middleware

● Grid middleware developedby the EU project EGEE-3 http://eu-egee.com/

● Different interacting components Compute Elements Storage Elements Authentication Services Accounting Services ...


Context: the gLite MiddlewareContext: the gLite Middleware

● Many of the middleware services are exposed as Web Services

● Standardization efforts are leading towards different interchangeable implementations of the same services

M. Riedel et al., Interoperation of World-Wide Production e-Science Infrastructures, Concurrency and Computation: Practice and Experience, 21(8), 2009

P. Andreetto et al., Standards-Based Job Management in Grid Systems, submitted to the Journal of Grid Computing


BPELBPEL

● The Service-oriented Architecture (SOA) paradigm foresees the creation of business applications from independently developed services.

● Providers offer similar competing services these offerings can differ significantly in some

QoS attributes like performance.● The key point is to build applications through

the composition of available services● The application can be specified using BPEL

notation


BPEL / 1BPEL / 1<receive>The executing process waits for a specific incoming message to be received;

<reply>

Sends a message in reply to a message which was received through a receive tag;<invoke> Invokes a one-way or request-response operation on a partner;

<wait>Waits for a given time period, or until a certain time has passed;

<sequence> Denotes a set of activities which should be executed sequentially; for each activity it is possible to specify additional dependencies, that is, other activities which must complete before executing the current one.

<If> Selects one activity from a set of choices;

<while> Repeats an activity until a certain predicate is no longer true;


BPEL / 2BPEL / 2

<repeatUntil>Repeats an activity until a condition becomes true;

<forEach> This activity repeats its child activity for a number N of times; the child activity instances can be executed sequentially, or in parallel;

<pick>The process blocks until a certain message is received, or a timeout goes off. When one of these events occurs, the associated activity is executed and the pick completes;

<flow> Denotes a set of concurrent activities;

<switch> Allows the process to choose exactly one branch


BPEL performance modellingBPEL performance modelling






AnnotatedAnnotatedWSDL and BPELWSDL and BPEL



QN Model QN Model EvaluationEvaluation

(bound analysis)(bound analysis)

BPEL2qnboundMarzolla and Mirandola, proc. QoSA'07


BPEL model to QN modelBPEL model to QN model


Example / BPELExample / BPEL<bpws:process> <perf:workload type="closed" thinktime="120"/> <bpws:import importType="http://schemas.xmlsoap.org/wsdl/" location="CaseStudy.wsdl"> <bpws:sequence> <bpws:invoke operation="Authenticate"/> <bpws:flow> <bpws:invoke operation="TransferISB"/> <bpws:invoke operation="TransferData"/> </bpws:flow> <bpws:while> <bpws:condition perf:prob="0.7"/> <bpws:invoke operation="JobStart"/> </bpws:while> <bpws:invoke operation="TransferOSB"/> <bpws:invoke operation="Analyze"/> </bpws:sequence></bpws:process>

Closed Workload

External Delay (think time)

Branch probability


Example / WSDLExample / WSDL

<definitions> <portType name="DataFactory"> <operation name="TransferData"> <perf:PAdemand resource="DF:CPU" value="1"/> <perf:PAdemand resource="DF:Disk" value="120"/> <perf:PAdemand resource="Network" value="80"/>

... </operation> </portType></definitions>

Operation Name

Resource NameAverage

Service Time


Example / WSDLExample / WSDL<definitions> <portType name="JobFactory"> <operation name="Authenticate"> <perf:PAdemand resource="CE:CPU" value="10"/> </operation> <operation name="TransferISB"> <perf:PAdemand resource="CE:CPU" value="2"/> <perf:PAdemand resource="Network" value="10"/> <perf:PAdemand resource="CE:Disk" value="120"/> </operation> <operation name="JobStart"> <perf:PAdemand resource="CE:CPU" value="4"/> </operation> <operation name="TransferOSB"> <perf:PAdemand resource="CE:CPU" value="1"/> <perf:PAdemand resource="Network" value="80"/> <perf:PAdemand resource="CE:Disk" value="30"/> </operation> </portType></definitions>


Example / WSDLExample / WSDL

<definitions> <portType name="DataAnalysis"> <operation name="Analyze"> <perf:PAdemand resource="DA:CPU" value="100"/> <perf:PAdemand resource="DA:Disk" value="30"/> </operation> </portType></definitions>


BPEL2qnboundBPEL2qnbound● Command-line tool● Given annotated BPEL and WSDL, we compute the

visit counts, and then the service demands

Bottleneck


BPEL2qnboundBPEL2qnbound

● We can study the effect of the bottleneck removal


Performance Evaluation of the Performance Evaluation of the Cell ProcessorCell Processor


The Cell/BEThe Cell/BE● Developed by

IBM/Sony/Toshiba Used inside Sony's

Playstation®3● Power Processor Element

PowerPC architecture● Synergistic Processor

Elements 128 registers, each

128bit wide 256KB local store Efficient vector

processinghttp://domino.research.ibm.com/comm/research.nsf/pages/r.arch.innovation.html


Cell ArchitectureCell Architecture


ChallengesChallenges

● Irregular architecture SPEs and PPE need different compilers

● Non-shared memory DMA transfer main memory ↔ SPE local stores

● Difficult to get it right Overlap SPE computations with DMA transfers Vectorize SPE code Data alignment and packing for efficient DMA

transfer/vector manipulation


Cell performance modelingCell performance modeling







QueueingQueueingNetworkNetworkModel (?)Model (?)


Model Model EvaluationEvaluation


B1

B2

B1

B2

Example: MultibufferingExample: Multibuffering● Fetch/Compute

Memory SPE

Proc. B1

Proc. B2

Proc. B1

Memory SPE

● Double-buffering: computations overlapped with fetch of next input buffer


QN model of the Cell/BEQN model of the Cell/BEMultibufferingMultibuffering

S1 S2

SPEMem

Multibuffering level = population size N

S1 = Mean DMA transfer time

S2 = Mean SPE computation time


ConclusionsConclusions

● Model-based system performance evaluation can be applied at different levels, in different scenarios From Software Systems... ...to Web Services... ...down to CPU programming

● Results are promising, BUT... Tools are lacking Model-based PE is poorly integrated into CASE

tools Difficult to find meaningful case studies


Thank you!Thank you!

http://www.phdcomics.com/comics/archive.php?comicid=942

Documents

Performance Evaluation of Complex Systems: from Large Software