Upload
ferdinand-paul
View
215
Download
2
Tags:
Embed Size (px)
Citation preview
SOFTWARE SYSTEM PERFORMANCE
CS 360
Lecture 11
PERFORMANCE OF COMPUTER SYSTEMSIn most computer systems:
The cost of people (development) is much greater than the cost of hardware
Yet, performance is importantA single bottleneck can slow down an entire systemFuture loads may be much greater than predicted
2
WHEN PERFORMANCE MATTERS Real time systems
when computation must be fast enough to support the service provided, e.g., fly-by-wire control systems have tight response time requirements.
Very large computations where elapsed time may be measured in days, e.g., calculation of weather
forecasts must be fast enough for the forecasts to be useful.
User interfaces where humans have high expectations, e.g., mouse tracking must appear
instantaneous.
Transaction processing where staff need to be productive and customers not annoyed by delays,
e.g., airline check-in. 3
HIGH-PERFORMANCE COMPUTINGHigh-performance computing:
Large data collections (e.g., Amazon) Huge numbers of users (e.g., Google) Large computations (e.g., weather forecasting)
Must balance cost of hardware against cost of so(ware development Some configurations are very difficult to program and debug Sometimes it is possible to isolate applications programmers from the system complexities
4
PERFORMANCE CHALLENGES FOR ALL SOFTWARE SYSTEMSTasks
Predict performance problems before a system is created. Design and build a system that is not vulnerable to performance problems.
Identify causes and fix problems after a system is implemented.
5
INTERACTIONS BETWEEN HARDWARE AND SOFTWAREExamples
In a distributed system, what messages pass between nodes? How many times must the system read from disk for a single transaction?
What buffering and caching is used? Are operations in parallel or sequential? Are other systems competing for a shared resource (e.g., a network or server farm)?
How does the operating system schedule tasks?
6
LOOK FOR BOTTLENECKSUsually, CPU performance is not the limiting factor.
Hardware bottlenecks Reading data from disk Shortage of memory (including paging) Moving data from memory to CPU Network capacity (bandwidth)
Inefficient software Algorithms that do not scale well Parallel and sequential processing
7
LOOK FOR BOTTLENECKSCPU performance is a limi1ng constraint in certain domains: large data analysis (e.g., searching) mathematical computation (e.g., engineering) compression and encryption multimedia (e.g., video) perception (e.g., image processing)
8
TIMESCALE OF DIFFERENT COMPONENTS
9
Actual performance may be considerably less than theoretical peak performance
LOOK FOR BOTTLENECKS: UTILIZATIONUtilization is the proportion of the capacity of a service that is used on average. Utilization =
avg. service time for a transaction/avg. arrival time of transactions
Peak loads and temporary increases in demand can be much greater than the average.
10
PREDICTING SYSTEM PERFORMANCEDirect measurement on subsystem (benchmark)
Disk I/O CPU Network
Simulation
All require detailed understanding of the interaction between software and hardware systems.
11
DIRECT MEASUREMENT ON SUBSYSTEMTools for examining performance of subsystems:
Ex: Sysbench
Sysbench is a modular, cross-platform, multi-threaded benchmark tool used to quickly gain information about system performance. Tests the following:
File I/O performance Job scheduler performance Memory allocation and transfer speed Database service performance 12
SIMULATIONSBuild a computer program that models the system as set of states and events
advance simulated time determine which events occurred update state and event list
repeat
Discrete time simulation: Time is advanced in fixed steps (e.g., 1 ms)
Next event simulation: Time is advanced to next event
13
MEASUREMENTS ON OS/ARCHITECTURESBenchmarks:
Run tools on standard problem sets, sample inputs, or simulated loads on the system
Collect data on resource availability, utilization, performance, etc.
Example: Disk I/O benchmark using MySQL to determine performance of different cloud stacks (OpenStack, Eucalyptus, Docker, etc.) Network, base OS, hardware should remain the same across all tests.
14
EXAMPLE: DISTRIBUTED SYSTEM
15Benchmark: Throughput vs. number of CPU cores utilized
FIXING BAD PERFORMANCE If a system performs badly, begin by identifying the cause: Instrumentation:
Add timers to the code. Often this will reveal that delays are centered in a specific part of the system.
Test loads: Run the system with varying loads (high transaction rates, large input files, many
users, etc.). This may reveal the characteristics of when the system runs badly.
Design and code reviews: Team review of system design, program design, and suspect sections of code. This
may reveal an algorithm that is running very slowly, e.g., a sort, locking procedure, etc.
Find the underlying cause and fix it or the problem will return! 16
PREDICTING PERFORMANCE CHANGE: MOORE’S LAWOriginal version:
The density of transistors in an integrated circuit will double every year. Gordon Moore, Intel, 1965
Current version: Cost/performance of silicon chips doubles every 18 months.
17
MOORE’S LAW: RULES OF THUMB
Silicon chips: cost/performance improves 30% / year in 12 years = 20x in 24 years = 500x
Storage media: cost/performance improves 40% / year in 12 years = 50x in 24 years = 3,000x
These assumptions are conservative. During some periods, the increases have been considerably faster.
Note: Recently, the rate of performance increase in individual components, such as CPUs, has slowed down, but the overall rate of increase has been maintained by placing many CPU cores on a single chip.
18
MOORE’S LAW EXAMPLE Will this be a typical laptop?
Surely there will be some fundamental changes in how these resources are packaged and used.
19
PARKINSON’S LAWOriginal:
Work expands to fill the time available. C. Northcote Parkinson
Software development version: Demand will expand to use all the hardware available. Low prices will create new demands. Your software will be used on equipment that you have not envisioned.
20
FALSE ASSUMPTIONS FROM THE PAST
Be careful about the assumptions that you make Here are some past assump1ons that caused problems: Unix file system will never exceed 2 GB. AppleTalk networks will never have more than 256 hosts (28 bits).
GPS software will not last more than 1024 weeks. Two bits are sufficient to represent a year (Y2K bug). Etc., etc., .....
21
SOFTWARE PERFORMANCE REQUIREMENTS (DOCUMENTATION)
Performance specifications should, at minimum, cover the following concepts:In detail, what is the performance test scope?
What subsystems, interfaces, components, etc. are in and out of scope of the specific performance test?
For the user interfaces, how many concurrent users are expected for each?Specify peak vs. normal usage
What does the target system (hardware) look like?Specify all hardware configurations
22
SOFTWARE PERFORMANCE REQUIREMENTS (DOCUMENTATION)
Performance specifications should, at minimum, cover the following concepts:What is the Application Workload Mix for each component?Ex: 20% login, 40% search, 30% item select, 10% checkout
What is the System Workload Mix?What processes are running?
What are the time requirements for any/all processes?Specify peak vs. normal
23
SOFTWARE PERFORMANCE REQUIREMENTS (DOCUMENTATION)
The performance requirements should include test cases to demonstrate performance of Individual components, Interfaces, Groups of components, And the system as whole
24
SOFTWARE PERFORMANCE REQUIREMENTS (DOCUMENTATION)
Tasks needed for performance documentation:1. Configure the testing environment2. Initial test run – to check correctness of testing script3. Execute tests (repeatedly) 4. Record and analyze the results (Pass/Fail)
Investigate corrective action if test fails5. Re-tune components/interfaces/system and repeat process
25
SOFTWARE PERFORMANCE REQUIREMENTS (DOCUMENTATION)
Tasks needed for performance documentation:6. Gathering performance requirements7. Decide whether to use internal or external resources to perform
tests.8. Developing a plan to test for performance9. Specify testing tools10.Specify test data11.Develop proof of concept test scripts for each
component/interface
26