Ch4b- 2EE/CS/CPE 3760 - Computer OrganizationSeattle Pacific University

Performance metrics

• I’m concerned with how long it takes to run my program

• Chances are, that number isn’t published with the specs for the computer

• Standardized metrics

• Benchmarks (SPEC, etc.)

• Clock Frequency

• MIPS

Ch4b- 3EE/CS/CPE 3760 - Computer OrganizationSeattle Pacific University

Benchmarks

• Run a suite of benchmark programs, average the performance

• Benchmarks - programs thought to be representative of commonly-used programs

• Advantages

• Actually corresponds to execution time!

• Represents a wider range of programs• Disadvantages

• Are they running your program?

• Who picks the benchmarks? Be wary if the manufacturer does!

Ch4b- 4EE/CS/CPE 3760 - Computer OrganizationSeattle Pacific University

SPEC Benchmarks

• SPEC (System Performance Evaluation Cooperative) maintains a set of benchmark suites

• SPEC Web Page (www.spec.org)

• New tests use SPEC CPU2006

• CINT2006 - Performance on integer programs

• CFP2006 - Performance on floating-point programs

• Larger numbers indicate better performance

• Tests prior to 2006 used CPU 2000 or CPU95

Ch4b- 5EE/CS/CPE 3760 - Computer OrganizationSeattle Pacific University

CINT2000 Results for Various Processors

Clock Speed (GHz)

CIN

T2

000

Note: Results depend on Cache size, memory system, and motherboard

400

900

1400

1900

2400

2900

3400

0.5 1

1.5 2

2.5 3

3.5 4

Pentium 3

Pentium 4

P4 Extreme

Xeon

Athlon

Athlon 64

Opteron

Pmac G5

Athlon FX (DC)

Core Duo

Core 2 Duo

Ch4b- 6EE/CS/CPE 3760 - Computer OrganizationSeattle Pacific University

CFP2000 Results for Various Processors

Note: Results depend on Cache size, memory system, and motherboard

200

700

1200

1700

2200

2700

3200

0.5 1

1.5 2

2.5 3

3.5

Pentium 3

Pentium 4

P4 Extreme

Xeon

Athlon

Athlon 64

Opteron

Pmac G5

Athlon FX (DC)

Core Duo

Core 2 Duo

Clock Speed (GHz)

CF

P2

000

Ch4b- 7EE/CS/CPE 3760 - Computer OrganizationSeattle Pacific University

Alternative: Web Benchmarks

• SPEC benchmarks are good, but difficult to sort through

• It can take hours of work just to find out the best performing variety of a system

• Results are available only if manufacturer submits them

• Web Sources – PassMark, Tom’s Hardware Guide

• Track performance of Windows-based machines for the past few years

• PassMark (cpubenchmark.net) uses user-submitted results – Benchmark your system free and compare.

• Tom’s (tomshardware.com) does their own testing

Ch4b- 8EE/CS/CPE 3760 - Computer OrganizationSeattle Pacific University

Improving System Peformance

• Assume we’re running a program that spends 40% of its time accessing memory

• Now, we upgrade the processor from 1 GHz to 4 GHz

• How much faster does the program run? (4X faster?)

• We’ve reduced the time for 60% of the program by 4

• But we haven’t touched the memory access time

• New total = Old * (40% + (60% / 4)) = Old * (40% + 15%) = Old * 55% Not even twice as fast!

Ch4b- 9EE/CS/CPE 3760 - Computer OrganizationSeattle Pacific University

Amdahl’s Law

• Practical effect: “Make the common case fast”

• Corollary: “Forget about the rare case”

• New Execution time =Execution time affected by impr. + Unaffected Execution Time

Amount of Improvement

• Example: 70% of my execution time is done on integer ADDs, and 6% on floating point ADDs. Total execution time is 100 seconds.

• What’s the effect of making integer ADDs twice as fast?• New time = (100 * .70) / 2 + (100 * .30) = 35+30=65 seconds

• What’s the effect of making F.P. Adds twice as fast?

• New time = (100 * .06) / 2 + (100 * .94) = 3+94 = 97 seconds

Ch4b- 10EE/CS/CPE 3760 - Computer OrganizationSeattle Pacific University

Speedup

When analyzing a change to a system, we often look at speedup.

Speedup = Old Execution Time / New Execution Time

Speedup > 1 Improvement!Speedup = 1 No changeSpeedup < 1 UhOh!

Continuing example from last slide: Int. Adds 2X faster: Speedup = 100s/65s = 1.54 FP Adds 2X faster: Speedup = 100s/97s = 1.03

Ch4b- 11EE/CS/CPE 3760 - Computer OrganizationSeattle Pacific University

Performance Summary

• Execution time is the most important performance metric

• Basic formula for performance:

• Execution time = instructions * cycle time * CPI

• Amdahl’s law describes how making limited improvements affects the bottom line

• Only make improvements in areas that are commonly used

• Standard benchmarks help us to compare performance of various computers

• Beware of overly-simplified comparisons