Performance Evaluation Methods
for
P2P Overlays
Simula Research Laboratory, May 18th, 2009
Knut-Helge Vik Simula Research Laboratory and University of Oslo.
Trial Lecture, Knut-Helge Vik University of Oslo
Outline Introduction
Peer-to-peer (P2P) networks
Peer-to-peer overlays
Performance evaluation
Performance measures of a computer system
Performance measures of a computer network
Performance evaluation methods for P2P overlays
Simulation model
Analytical model
Measurements
Tradeoffs in performance evaluation
Conclusions and summaries
Trial Lecture, Knut-Helge Vik University of Oslo
*Stefan Schmid @ HiPC 2007
Many types of networks*…
DISTRIBUTED COMPUTING Internet Graph Web Graph Neuron Networks
Social Graphs Public Transportation Networks
Different properties: Natural vs. Man-made Robustness End-to-end latency Routability ...
Peer to peer networks
Trial Lecture, Knut-Helge Vik University of Oslo
Peer-to-peer networks: Peer: Somebody or something who/that is at an equal level
Peer-to-peer system is a network of clients that share resources – Bandwidth!
− File sharing, telephone, video-conference, etc.
Dozens of peer-to-peer applications, many are used to share copyrighted data-files
Peer-to-peer networks differ from client-server networks
− In peer-to-peer networks clients are communicating directly with each other
− Peer-to-peer networks are distributed around the world!
= Router
= Client
= Internet
Peer-to-peer Client-server
= Client = Server
Trial Lecture, Knut-Helge Vik University of Oslo
Peer-to-peer overlay networks
An overlay network is a logical network built on top of a physical network
Overlay links are tunnels through the underlying network
Many logical networks may coexist at once over the same underlying network and providing its own particular service
Nodes are often end hosts acting as intermediate nodes that forward traffic, providing a service, such as access to files
- P2P overlays are built on top of the Internet topology
Trial Lecture, Knut-Helge Vik University of Oslo 6
Peer-to-peer overlay networks
Trial Lecture, Knut-Helge Vik University of Oslo 7
Peer-to-peer overlay networks
Focus at the application level
Trial Lecture, Knut-Helge Vik University of Oslo
Peer-to-peer overlay networks
Unstructured
− Without prior knowledge of the topology
− Peers are organized according to optimization metrics (if any)
• Freenet
• Gnutella
• FastTrack/KaZaA
• BitTorrent
• Overnet/eDonkey
Structured
− Topology is tightly controlled
− DHT (distributed hash table) enables efficient lookup
• CAN
• Chord
• Tapestry
• Pastry
• Kademlia
• Viceroy
hybrid
Trial Lecture, Knut-Helge Vik University of Oslo
Outline Introduction
Peer-to-peer (P2P) networks
Peer-to-peer overlays
Performance evaluation
Performance measures of a computer system
Performance measures of a computer network
Performance evaluation methods for P2P overlays
Simulation model
Analytical model
Measurements
Tradeoffs in performance evaluation
Conclusions and summaries
Trial Lecture, Knut-Helge Vik University of Oslo
Performance evaluation in computer systems
Performance: Manner or quality of functioning, fulfillment of a claim, promise, request
Evaluation: Systematic determination of merit, worth, and significance of something or someone using criteria against a set of standards.
Scientific method: Describes the process going from observation to validated theory
System: a collection of entities (e.g., people and machines) that interact over time
Performance evaluation is the application of scientific methods to the study of computer systems [Bob Kinicki].
− We apply scientific methods when we do performance evaluation of computer systems
The goal of performance evaluation is to determine the effectiveness and fairness of a computer system that is assumed to work correctly [Bob Kinicki].
− Find out how well the computer system works based on certain criteria
The goal of computer systems engineers, scientists, analysts, and users is to get the highest performance for a given cost [Raj Jain].
− Best performance, but make it cheap
Trial Lecture, Knut-Helge Vik University of Oslo
Performance measures of a computer network
Metric: a descriptor used to represent some aspect of a computer network’s performance.
Need to measure basic characteristics of a computer network:
− Count how many times an event occurs
− Duration of some time interval
− Size of a parameter, etc.
These characteristics are called performance metrics
Focus is on system-oriented performance metrics – latency, bandwidth, etc.
− Quality of algorithms, protocols, methods
Add up metric values and calculate the average, mean, etc
− Make metrics understandable!
Metric statistics are used to visualize results
Trial Lecture, Knut-Helge Vik University of Oslo
Dos and don’ts of performance evaluation
Performance evaluation is often driven by the fact that we want the best!
Unclear goal? No model can solve everything.
Are you biased? Be fair when comparing systems.
Be systematic when comparing.
Understand the problem at hand.
Use the right performance metrics.
Use an appropriate level of detail in the evaluation.
Present results clearly.
State your assumptions.
Trial Lecture, Knut-Helge Vik University of Oslo
Outline Introduction
Peer-to-peer (P2P) networks
Peer-to-peer overlays
Performance evaluation
Performance measures of a computer system
Performance metrics of a computer network
Performance evaluation methods for P2P overlays
Simulation model
Analytical model
Measurements
Tradeoffs in performance evaluation
Conclusions and summaries
Trial Lecture, Knut-Helge Vik University of Oslo
Performance evaluation methods
In general, peer-to-peer network evaluation
makes use of the actual network, an emulated
network or a model of the network.
Model-based techniques
− Accessible system does not exist or is too large (complex, unhandy, not available, etc.)
− Instead, develop a model that mimics a system
Measurement-based techniques
− Accessible system exists
− Run system and evaluate its performance
− PlanetLab is a distributed network of computers used to conduct measurement studies
Trial Lecture, Knut-Helge Vik University of Oslo
Model-based performance evaluation
Model-based techniques:
− Simulation Modeling
− Analytical Modeling
− Both modeling techniques tend to rely on queuing theory.
Model-based techniques rely on simplifying assumptions that enable the model to capture important characteristics of networks
These models are usually networks of queues.
Arrivals
Queue Server
Trial Lecture, Knut-Helge Vik University of Oslo
Measurement-based performance evaluation Empirical: information gained by means of observation, experience, or
experiment
Empirical measurement of real/actual networks
P2P network measurements can be either active or passive.
Passive measurement studies account for existing traffic
− For example, collect information about packets traversing a network channel.
Active measurement involves purposely adding traffic to the network specifically to facilitate the measurement
− For example, sending packet train probes into the network to estimate the available bandwidth along a flow path.
Sender
Internet
Receiver Packet train
…
Analyze packets interarrival time and calculate available bandwidth
Trial Lecture, Knut-Helge Vik University of Oslo
Taxonomy of performance evaluation methods*
*Simulation, Modeling & Analysis (3/e) by Law and Kelton, 2000, p. 4, Figure 1.1
Trial Lecture, Knut-Helge Vik University of Oslo
Simulation models Event: a software message indicating that something has happened
Transition: passage from one state to another
A network simulator is a program that predicts the behavior of a network, without an actual network being present.
Event-driven simulation defines a P2P network in terms of states and transitions where events trigger transitions.
There are basically three types of simulators*[Lilja]:
− Emulation
− Static/Monte-carlo simulation
− Discrete-event simulation
*Other simulator types exist, but they are outside the scope
Event loop:
Trial Lecture, Knut-Helge Vik University of Oslo
Simulation models: Emulation
An emulator program is a simulation program that runs on some existing system to make that system appear to be something else.
Often used to provide a framework for testing new peer to peer systems.
Emulators can accurately evaluate how overlays behave when there is network traffic
ModelNet is a scalable wide-area network emulator:
− Deploy hundreds of instances of an application over nodes that behave as if they were distributed all over the world
− Widely used for peer-to-peer system evaluation
− De-facto standard for network emulation
− Unlike simulations, models cross traffic and packet loss
− Unlike PlanetLab, gives reproducible results
- Internet emulator nodes (FreeBSD) - Application emulator nodes (Linux)
Trial Lecture, Knut-Helge Vik University of Oslo
Simulation Models: Static/Monte Carlo
A static simulation has no time parameter
− Runs until some equilibrium state reached
Used to approximate the probability of certain outcomes by running multiple trial runs, called simulations, using random variables
Driven with random number generator
− So “Monte Carlo” (after casinos) simulation
Used to model physical phenomena, evaluate probabilistic system, numerically estimate complex mathematical expression
Peer to peer overlays: Often used to evaluate and simulate structured overlays[Beaumont].
Advantages: It is a good way of evaluating whether overlays have a chance of getting to undesired states
Drawbacks: No sense of time, we don’t know if it ends
Schematic showing the principal of stochastic uncertainty propagation. (The basic principle behind Monte Carlo simulation.)
Imagine throwing dart at square
− Random x (0,1)
− Random y (0,1)
Count if inside
− sqrt(x2+y2) < 1
Compute ratio R
− in / (in + out)
Can repeat as many times as needed to get arbitrary precision
Trial Lecture, Knut-Helge Vik University of Oslo
Simulation Models: Discrete-event
Simulate only at the interesting events
Often a simple way of solving a problem – simulate system and observe results
Discrete-event simulators are
− Stochastic = probabilistic
− Dynamic = changes over time
− Discrete = instantaneous events are separated by intervals of time
Time may be modeled in a variety of ways within the simulation.
− Time as linked events is most common:
Discrete-event simulators classified according to how they generate events[Lilja]*:
− Trace-driven -
− Distribution-driven –
*Other discrete-event simulators exist, but are outside the scope
Trial Lecture, Knut-Helge Vik University of Oslo
Discrete-event simulator: Trace-driven Trace: is a record of the sequence of events that occurred when the
system was traced
Application events are recorded in a trace and the trace is used to drive the discrete event simulator
Trace must be independent of the tested algorithms/protocols
Advantages:
− It is easier to sell than random numbers
− Easy validation, accurate workload, fair comparison, similarity to actual implementation
Drawbacks:
− Trace represents only one possible execution path of a system
− Complexity, finiteness , single point of validation
Often used to evaluate existing peer to peer applications
− Determine how the overlays evolve during execution
− Identify inefficiencies of the overlay construction in peer to peer application
− Many peer to peer applications have a naïve (random) overlay construction
Trial Lecture, Knut-Helge Vik University of Oslo
Discrete-event simulator: Distribution-driven
Input events are generated by the simulator, sequence of random numbers, etc,
− Common to apply models that generate events that mimic the intended behavior
No assurance that any real application program would produce this input sequence
− run many times with several different input sequences
Common approach for evaluating peer to peer overlays
− For example, in terms of graph theoretical metrics
− OMNet++, ns-2, GloMoSim, Ptolemy
Advantages: Control of the input events and the sequence makes it easier to implement
− Flexible, scalable, supports different event generators
Drawbacks: Harder to trust sequences of random numbers as the foundation of statistics
− Credibility, accuracy
Generic event loop
Ns-2 network
Trial Lecture, Knut-Helge Vik University of Oslo
Analytical models A stochastic process is one whose behavior is non-deterministic
Describe an application and the architecture using equations
Analytical models of computer networks tend to be stochastic models built on the theory of stochastic processes associated with independent random variables.
Queuing theory is a commonly used technique
Complexity analysis of algorithms, eliminate exponential time
Advantages:
− Fast & easy analysis of relations
− Allows extrapolation to ridiculous parameters, e.g. thousands of processors
Drawbacks:
− Sometimes infeasible to obtain accuracy (e.g. modelling caches)
− To obtain reasonable accuracy, the models may become very complex (e.g. modelling of network contention)
Peer-to-peer overlays: (complexity analysis) find the theoretical bounds for algorithms
Trial Lecture, Knut-Helge Vik University of Oslo
Measurement studies
Measurement Experiments are divided into two major categories:
− Live measurements (observation) − Controlled-traffic measurements
Trial Lecture, Knut-Helge Vik University of Oslo
Measurement studies: Live measurement
Live empirical studies measure the performance of the computer network while it is handling real traffic.
One advantage of this type of study is that the measurement involves a real workload.
A drawback of measuring live traffic is being convinced that this measurement involves ‘typical’ traffic for this network.
Another drawback of live traffic measurement is that reproducibility of the exact same traffic workload is usually not possible.
PlanetLab is used for live measurement studies of peer to peer overlays
Issues with trustworthiness, but it is the largest real test-network around
Trial Lecture, Knut-Helge Vik University of Oslo
Measurement studies: Controlled traffic Use traffic generator tools or script files to
provide repeatable, controlled traffic workloads on the network being measured.
Chosen when the goal is to evaluate the impact of different versions of a network component, strategy or algorithm on network performance.
Advantage: Controlled, repeatable traffic makes it easier to conduct cause-and-effect performance analysis.
Drawback: the accuracy of the traffic generator tool, and the ability to conduct measurement experiments where the traffic workload choices are adequately varied to provide representative, robust network performance evaluation.
PlanetLab is also used for controlled traffic
Trial Lecture, Knut-Helge Vik University of Oslo
Trade-offs in performance evaluation
Trade-off: losing one quality of something in return for gaining another quality.
Analytical models
− Model includes assumptions that may be to tough
− Little or no extra equipment, other than ordinary computers
− May reach eliminating conclusions fairly fast – complexity analysis
Simulation models
− Accuracy and computational intensity
− High vs. low level of detail, often means slow vs. fast
Measurement models
− Tougher to implement, but increases the credibility of the evaluation
Performance evaluations that combine results from analytical, simulation and measurement based studies, often reach more precise and credible evaluations.
How do we compare results from different models?
A common evaluation framework makes the results directly comparable
Trial Lecture, Knut-Helge Vik University of Oslo
Comparisons of performance evaluation methods
Comparison of Performance Evaluation Techniques [Raj Jain]
Trial Lecture, Knut-Helge Vik University of Oslo
Conclusions and summaries
Peer to peer overlays can be evaluated using
− Simulation models
− Analytical models
− Measurements
A common evaluation framework makes the results directly comparable
Performance evaluations that combine results from analytical, simulation and measurement based studies, often reach more precise and credible evaluations.
Questions?
Trial Lecture, Knut-Helge Vik University of Oslo
References
[Bob Kinicki] Performance Evaluation of Computer Networks, Presentation, Worcester Polytecnic Institute, 2006
[Willig04] Performance Evaluation Techniques Summer 2004, Dr.-Ing. Andreas Willig, Hasso-Plattner-Institut, Universitat Potsdam, [email protected], April 25, 2005
[Raj Jain], "The Art of Computer Systems Performance Analysis -- Techniques for Experimental Design, Measurement, Simulation and Modeling", Wiley, 1991
Averill M. Law and W. David Kelton, "Simulation Modeling and Analysis, 3rd Edition", McGraw-Hill, 2000
Jerry Banks and John S. Carson and Barry L. Nelson and David M. Nicol, "Discrete-Event System Simulation", Prentice-Hall, 2000
Reuven Y. Rubinstein and Benjamin Melamed, "Modern Simulation and Modeling", Wiley, 1998
[Beaumont] Peer to Peer Multidimensional Overlays: Approximating Complex Structures, Olivier Beaumont, Anne-Marie Kermarrec, and Étienne Rivière, 2007
[Dinh] Large scale distributed simulation of p2p networks, TTA Dinh, M Lees, G Theodoropoulos, R Minson, 2008
[ModelNet] http://issg.cs.duke.edu/modelnet.html