1

Industry Standard Benchmarks: Past, Present and Future Raghunath Nambiar Distinguished Engineer, Cisco RNambiar@cisco.com

Invited Talk

2

• Cisco Distinguished Engineer, Chief Architect, Big Data Solutions, Cisco

• General Chair, TPC’s International Conference Series on Performance Evaluation and Benchmarking (TPCTC)

• Chairman, TPC Big Data Committee

• Industry Chair, IEEE Big Data 2013, ICPE 2014

• Board Member TPC, WBDB, BigDataTop100

3

4

• Synthetic Benchmarks

Simulate functions that yield an indicative measure of the subsystem performance

Widely adapted in the industry and academic community

Several open source tools

• Application Benchmarks

Developed and administered by application vendors

VMmark, SAP and Oracle application benchmarks

• Industry Standard Benchmarks

Driven by industry standard consortia which are represented by vendors, customers, and research organizations

Democratic procedures for all key decision making

TPC, SPEC and SPC

5

Better products,

Lower Price-

Performance

• Industry standard benchmarks have played, and continue to play, a crucial role in the advancement of the computing industry

• Demands for them have existed since buyers were first confronted with the choice between purchasing one system over another

• Historically we have seen that industry standard benchmarks enable healthy competition that results in product improvements and the evolution of brand new technologies

6

Critical to Vendors, Customers and Researchers

• Vendor

Demonstrate competitiveness of their products

Monitor release-to-release progress of their products under development

• Customer

Cross-vendor evaluation of technologies and products in terms of performance, price-performance, energy efficiency

• Researcher

Known, measurable, and repeatable workloads to develop and enhance relevant technologies

7

Benchmark Development Process Major Activities

• Development of new benchmarks

• Publication of benchmark results

• Refinement of existing benchmarks

• Resolution of disputes and challenges

Source: Raghunath Nambiar, Meikel Poess: The Making of TPC-DS. VLDB

2006: 1049-1058

8

• The TPC is a non-profit, vendor-neutral organization, established in August 1988

• Reputation of providing the most credible performance results to the industry.

• Role of “consumer reports” for the computing industry

• Solid foundation for complete system-level performance

• Methodology for calculating total-system-price and price-performance

• Methodology for measuring energy efficiency of complete system

Source: Raghunath Nambiar, Matthew Lanken, Nicholas Wakou, Forrest Carman, Michael Majdalany: Transaction Processing Performance Council

(TPC): Twenty Years Later - A Look Back, a Look Ahead, First TPC Technology Conference, TPCTC 2009, Lyon, France, ISBN 978-3-642-10423-7

9

TPC-A

TPC-B

TPC-C

TPC-D

TPC-R

TPC-H

TPC-W

TPC-App

TPC-E

TPC-DS

TPC-VMS

Pricing

Energy

TPC-DI

TPC-VMC

TPC-V

Common Specifications

Developments in Progress

Benchmark Standards

• Obsolete

• Active

• Common Specifications

• In Progress

Source: Raghunath Nambiar, Meikel Poess, Andrew Masland, H. Reza Taheri, Matthew Emmerton, Forrest Carman,

Michael Majdalany: TPC Benchmark Roadmap 2012, 4th TPC Technology Conference, TPCTC 2012, Istanbul, Turkey,

ISBN 978-3-642-36726-7

• Developed 11 Benchmark Standards

• 5 Standards are current

• What’s new ? • TPC-VMS – new standard for measuring database

performance in a virtualized environment

• TPC-DI - standard for measuring data integration

performance. Expected to be standard in 2014

• TPC-Big Data committee was formed in October 2013

10

Universities and research organizations are encouraged to join the TPC as Associate

Members.

To join the TPC: http://www.tpc.org/information/about/join.asp

11

• The Standard Performance Evaluation Corporation (SPEC) is a non-profit organization established in 1988

• Develop standards for system level performance measurements

• History of developing relevant benchmarks to the industry in a timely manner

• Four diverse groups - Graphics and Workstation Performance Group (GWPG), High Performance Group (HPG), Open Systems Group (OSG) and Research Group (RG)

• Represented by system and software vendors and a number of academic and research organizations

12

• SPEC CPU2006 is designed to measure the compute power of systems, contains two

benchmark suites: CINT2006 for measuring and comparing compute-intensive integer performance, and CFP2006 for measuring and comparing compute-intensive floating point performance

• SPEC MPI2007 is designed for evaluating MPI-parallel, floating point, and compute-intensive performance across a wide range of cluster and SMP hardware

• SPECjbb2013 measures server performance based on the Java application features

by emulating a three-tier client/server system

• SPECjEnterprise2010 measures system performance for Java Enterprise Edition based application servers, databases, and supporting infrastructure

• SPECsfs2008 is designed to evaluate the speed and request-handling capabilities of file servers utilizing the NFSv3 and CIFS protocols

• SPECpower_ssj2008 evaluates the power and performance characteristics of volume server class computers

• SPECvirt_sc2010 measures the end-to-end performance of all system components,

including the hardware, virtualization platform, virtualized guest operating system, and application software

13

• The Storage Performance Council (SPC) is a vendor-neutral consortia established in 2000

• Focused on industry standards for storage system performance

• Serve as a catalyst for performance improvement in storage subsystems

• Robust methodology measuring, auditing, and publishing performance, price-performance, and energy-efficiency metrics for storage systems

• Major systems and storage vendors are members of the SPC

14

• SPC Benchmark 1 (SPC-1) consists of a single workload designed to demonstrate the performance of a storage

subsystem under OLTP workloads characterized by random reads and writes

• SPC Benchmark 1/Energy (SPC-1/E) is an extension of SPC-1 that consists of the complete set of SPC-1

performance measurement and reporting plus the measurement and reporting of energy consumption

• SPC Benchmark 2 (SPC-2) consists of three distinct workloads: large file processing, large database queries, and

video on-demand simulating the concurrent large-scale sequential movement of data

• SPC Benchmark 2/Energy (SPC-2/E) is extension of SPC-2 that consists of the complete set of SPC-2

performance measurement and reporting plus the measurement and reporting of energy consumption

• SPC Benchmark 1C (SPC-1C) is based on SPC-1 for storage component products such as disk drives, host bus

adapters, storage enclosures, and storage software stacks such as volume managers

• SPC Benchmark 1C/Energy (SPC-1C/E) is an extension of SPEC-1C that consists of the complete set of SPC-1C

performance measurements and reporting plus measurement and reporting of energy consumption

• SPC Benchmark 2C (SPC-2C) is based on SPC-2, predominately by large I/Os organized into one or more

concurrent sequential patterns for storage component products

• SPC Benchmark 2C/Energy (SPC-2C/E) is an extension of SPC-2C that consists of the complete set of SPC-2

performance measurement and reporting plus the measurement and reporting of energy consumption

15

TPC-C Performance 1992-2010

100

1000

10000

100000

1000000

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

Av

erag

e tp

mC

/Pro

cess

or

Publication Year

Average tpmC per processor

Moore's Law

TPC-C Revision 2

TPC-C Revision 3, First Windows result and first x86 result

First clustered result

First result using 7.2K RPM disk drives First storage area network (SAN) based result

Intel introduces multi threading

TPC-C Revision 5 and First x86-64 bit result

First multi core result

First Linux result

First result using 15K RPM disk drives

First result using 15K RPM SAS SFF disk drives

First result using solid state drives (SDD)

Source: Nambiar R., Poess M. (2010). Transaction Performance vs. Moore’s Law. Performance Evaluation, Measurement and Characterization of

Complex Systems. Lecture Notes in Computer Science 6417, Springer 2011, ISBN 978-3-642-18205-1

16

TPC-C Price-Performance 1992-2010

0.1

1

10

100

1000

10000

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

Pri

ce p

er N

tpm

C

Publication Year

Price per NtpmC Moore's Law

TPC-C Revision 2

TPC-C Revision 3, First Windows result and first x86 result First clustered result

First result using 7.2 RPM disk drives

First storage area network (SAN) based result

TPC-C Revision 5 and First x86-64 bit result

Intel introduces multi threading First multi core result

First Linux result

First result using 15 K RPM disk drives

First result using 15K RPM SAS SFF disk drives

First result using solid state drives SDD

Source: Nambiar R., Poess M. (2010). Transaction Performance vs. Moore’s Law. Performance Evaluation, Measurement and Characterization of

Complex Systems. Lecture Notes in Computer Science 6417, Springer 2011, ISBN 978-3-642-18205-1

17

• IT 1.0: 1980-2000 Transaction Processing, Data Warehousing, File

server, Web server, Multi-tier Applications

• IT 2.0: 2000-2010 Internet centric, Massive scale-out systems,

Virtualization, Energy efficient systems

• IT 3.0: 2010- Cloud, Big Data, Internet of things, Software

defined and application centric infrastructure

Industry

Standard

Committees

have done a

great job

Call for new

standards

18

19

34.3% of World’s Population

have internet access today

50% by 2020

20

. . . . . .

There are 15 billion devices connected to the Internet

that’s 2.2 devices for every man, woman, and child on the planet earth

50 Billion devices by 2020

Trillion+ with IOT.

0

20

40

60

20122020

15 Billion

50 Billion

Connected Devices

Source: Cisco, webpronews.com

21 Source: time.com

Time’s Man of the Year

1982

22

If

Were a country …

. . . . . .

1. China (1.339 billion)

2. India (1.218 billion)

3. Facebook (1 billion)

4. United States (311 million)

5. Indonesia (237 million)

6. Brazil (190 billion)

7. Pakistan (175 million)

8. Nigeria (158 million)

9. Bangladesh (150 million)

10. Russia (142 million)

23

The third generation of IT platform driven by new applications and services built on cloud, mobile devices, social media, IoTs and more

Examples: Recommendation engines, Personalized contents, Crowd-sourcing

Source: IDC

Social Business

Mobile Broadband

Big Data/ Analytics

Cloud Services

Mobile Devices

and Apps

2011

1986

Millions of Users

Thousands of Apps

Hundreds of Millions of Users

Tens of Thousands of Apps

Millions of Apps

Billions of Users

Trillions of “Things”

LAN/ Internet

Client- Server

PC

Intelligent Economy

24

How many disk drives were sold in 2012 ?

1 Zettabyte = 1 099 511 627 776 Gigabytes

= 1 Billion 1TB Disk Drives

2008

0.5 Zettabyte

2011

2.5 Zettabytes

2020

35 Zettabytes

Source: IDC, EMC

25

616EB

Per Capita IP Traffic Per Capita

Internet Traffic

Source: Cisco

In 2016, equivalent of all movies ever made will cross global IP networks every 3 minutes

Global IP Traffic

Per Capita Internet Traffic

26

27

• Big Data is becoming integral part of IT ecosystem across all major verticals

• One of the most talked about topics in research and government sectors

• Big Data challenges can be summed up in 5V’s: Volume, Velocity, Varity, Value, Veracity

• Big Data is becoming center of 3I’s: Investments, Innovation, Improvization*

* Source: http://blogs.cisco.com/datacenter/ieee-bigdata/

Source: Mckinsey Global Institute Analysis

Source: Gartner 2011

Source: Cisco

29

• TeraSort

• YCSB

• GridMiX

• HiBench

• TPC-DS (at large scale ?)

• BigBench, BigDataBench

30

State of the Nature - early 1980's

the industry began a race that has accelerated over time: automation of daily end-user

business transactions. The first application that received wide-spread focus was

automated teller transactions (ATM), but we've seen this automation trend ripple through

almost every area of business, from grocery stores to gas stations. As opposed to the

batch-computing model that dominated the industry in the 1960's and 1970's, this new

online model of computing had relatively unsophisticated clerks and consumers directly

conducting simple update transactions against an on-line database system. Thus, the on-

line transaction processing industry was born, an industry that now represents billions of

dollars in annual sales.

Early Attempts at Civilized Competition

In the April 1, 1985 issue of Datamation, Jim Gray in collaboration with 24 others from

academy and industry, published (anonymously) an article titled, "A Measure of

Transaction Processing Power." This article outlined a test for on-line transaction

processing which was given the title of "DebitCredit." Unlike the TP1 benchmark, Gray's

DebitCredit benchmark specified a true system-level benchmark where the network and

user interaction components of the workload were included. In addition, it outlined several

other key features of the benchmarking process that were later incorporated into the TPC

process:

The TPC Lays Down the Law

While Gray's DebitCredit ideas were widely praised by industry opinion makers, the

DebitCredit benchmark had the same success in curbing bad benchmarking as the

prohibition did in stopping excessive drinking. In fact, according to industry analysts like

Omri Serlin, the situation only got worse. Without a standards body to supervise the

testing and publishing, vendors began to publish extraordinary marketing claims on both

TP1 and DebitCredit. They often deleted key requirements in DebitCredit to improve their

performance results.

From 1985 through 1988, vendors used TP1 and DebitCredit--or their own interpretation

of these benchmarks--to muddy the already murky performance waters. Omri Serlin had

had enough. He spearheaded a campaign to see if this mess could be straightened out.

On August 10, 1988, Serlin had successfully convinced eight companies to form the

Transaction Processing Performance Council (TPC).

31

• Performance

• Cost of ownership

• Energy efficiency

• Floor space efficiency

• Manageability

• User experience

32

• Relevant

• Repeatable

• Understandable

• Fair

• Verifiable

• Economical

Reference: K. Huppler, The Art of Building a Good Benchmark, Performance Evaluation and Benchmarking, LNCS vol. 5895, Springer 2009

• Time to Market – Long development cycle is not acceptable

33

• Business Case

• Data Definition and Data Generation

• Workload

• Execution Rules

• Metric

• Audit Rules

• Full Disclosure Report

34

• TPC International Technology Conference Series on Performance Evaluation and Benchmarking (TPCTC)

• Workshop Series on Big Data Benchmarking (WBDB)

• TPC - Big Data Benchmark Work Group (TPC-BD)

35

TPC International Technology Conference Series on Performance Evaluation and Benchmarking (TPCTC)

Accelerate the development of relevant benchmark standards

Enable collaboration between industry experts and researchers

Collocated with International Conference on Very Large Data Bases (VLDB) since 2009

TPCTC 2009 in conjunction with VLDB 2009, Lyon, France

TPCTC 2010 in conjunction with VLDB 2010, Singapore

TPCTC 2011 in conjunction with VLDB 2011, Seattle, Washington

TPCTC 2012 in conjunction with VLDB 2012, Istanbul, Turkey

TPCTC 2013 in conjunction with VLDB 2013, Riva Del Garda, Italy

TPCTC 2014 – will collocate with VLDB 2014 in Hangzhou, China (More information available at http://www.tpc.org/tpctc/

36

Workshop Series on Big Data Benchmarking (WBDB)

• A first important step towards the development of a set of benchmarks for providing objective measures of the effectiveness of hardware and software systems dealing with Big Data applications

• Open forum for discussions issues related to Big Data benchmarking

• WBDB Workshops WBDB 2012, San Jose

WBDB 2012.in, Pune

WBDB 2013.cn, Xi’an

WBDB 2013, San Jose

WBDB 2014, Potsdam, Germany (August 5-6, 2014)

• BigData100

37

38

• Evaluate big data workload(s) and make recommendations

• Four workloads under evaluation

• Additional workloads will be considered

• Accept one or more benchmarks to address various Big Data use cases

• More information: http://www.tpc.org/tpcbd/

39

• TPC is an international organization. Vendors, Customers, Universities, and Research Institutions are invited to join

• Membership Benefits Influence in the TPC benchmarking development process

Timely access to ongoing proceedings

Product Improvement

• Memberships Full membership - Participate in all aspects of the TPC's work, including development

of benchmark standards and setting strategic direction.

Associate Membership – Reserved for non-profit, educational institutions, market researchers, publishers, consultants, governments, businesses who do not create, market or sell computer products or services.

Promotional membership for new members

• More Information: http://www.tpc.org/information/about/join.asp

Thank you.