BigDataBench First Part-1213 - Big Data and AI Benchmark...

BigDataBench Tutorial

Jianfeng Zhan, Zhen Jia, and Gang Lu

http://prof.ict.ac.cn/BigDataBench

TE OF C

Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese

Academy of Sciences

BigDataBench TutorialMICRO 2014 Cambridge, UK

AcknowledgementsAcknowledgements

BigDataBench contributorsLei Wang, Chunjie Luo, Zhen Jia, Wanling Gao, Dr. g, j , , g ,Rui Han, Dr. Yuqing Zhu, Qiang Yang, XinlongLin, Jingwei Li, Wei Zhu, g ,Shujie Zhang, Dr. Chuliang Weng Dr Yongqiang HeDr. Yongqiang HeXiaona Li Bizhu QiuKent Zhan, Zijian Ming

BigDataBench MICRO 2014

Acknowledgements (Cont’)Acknowledgements (Cont )

Great thanks for Prof. Jason Mars to invite us to give this tutorial at Micro’14. g

BigDataBench Tutorial Program (1)BigDataBench Tutorial Program (1)

9:00‐9:40 Jianfeng ZhanWhat is BigDataBench? gBigDataBench benchmarking methodology

9 40 10 00 G L9:40‐10:00 Gang LuBigDataBench data sets and workloads

10:00‐10:30 Coffee break

BigDataBench Tutorial Program (2)BigDataBench Tutorial Program (2)

10:30‐11:00 Gang LuHow to use BigDataBench data sets and gworkloads?How to generate Large‐scale data sets?How to generate Large scale data sets?Multi‐tenancy version of BigDataBench

11:00‐12:00 Zhen Jia BigDataBench subsettingg gHow to use the simulator versions of BigDataBench ?

BigDataBench ?

Handbook of BigDataBenchHandbook of BigDataBench

Please feel free to download and distribute this handbook.

http://prof.ict.ac.cn/BigDataBench_micro_14/Draft versionDraft version93 pages

BigDataBench handbook: 1st partBigDataBench handbook: 1st part

Section 1 Summary of BigDataBench 3.1Section 2 Benchmarking methodologySection 2 Benchmarking methodologySection 3 BigDataBench specificationSection 4 BigDataBench implementationsSection 5 BigDataBench subsettingSection 5 BigDataBench subsetting

BigDataBench handbook: 2th partBigDataBench handbook: 2th part

Section 6 BigDataBench simulator versionSection 7 Multi‐tenancy version ofSection 7 Multi tenancy version of BigDataBenchS i 8 U lSection 8 User manual Section 9 BigDataBench usersgSection 10 Q&A

OutlineOutline

What is BigDataBench?

BigDataBench benchmarking methodology

Why Big Data Benchmarking?Why Big Data Benchmarking?

Measuring big data systems and architectures quantitatively

quantitatively

What is BigDataBench?What is BigDataBench? An open source big data benchmarking project p g g p j• http://prof.ict.ac.cn/BigDataBench

• Search Google using “BigDataBench”

What is BigDataBench (cont’)?g ( )BDGS(Big Data Generator Suite) for scalable data

Facebook Social Network

ImageNet

Wikipedia Entries

E‐commerce Transaction

English broadcasting audio

Amazon Movie Reviews

ProfSearch Resumes

DVD Input Streams

Google Web Graph

SoGou Data

Image scene

Genome sequence data Assembly of the human genome

ImpalaNoSql

14 Real‐world Data Sets

Impala

Search Engine

H d RDMA

SocialNetwork

E-commerce

Software Stacks33 Workloads

DataMPI

Hadoop RDMAMultimedia Bioinformatics

Software Stacks33 Workloads

BigDataBench evolutionBigDataBench evolution

5 application domains: 14 data sets and 33 workloadsSame specifications: diverse implementationsMulti‐tenancy version

h b d l

BigDataBench 3.1

Multidisciplinary effort

BigDataBench subset and simulator version 2014.12

BigDataBench 3.0

Typical Internet service domains

2014.4p y

32 workloads: diverse implementations

BigDataBench 2.0

2013.12

Typical Internet service domainsAn architectural perspective19 workloads & data generation tools

CloudRank 1.0DCBench 1.0BigDataBench 1.02013.7

Search engine6 workloads

11 data analytics workloads

Mixed data analytics workloads

Why BigDataBench?Why BigDataBench?Specification

Application domains

Workload Types

Workloads

Scalable data sets (from real

Multipleimpleme

Multitenancy

Subsets

Simulatorca o do a s ypes oads se s ( o ea

data)p e e

ntationsa cy e s o

version

BigDataBench Y Five Four 33 8 Y Y Y Y

BigBench Y One Three 10 3 N N N NCloudSuite N N/A Two 8 3 N N N Y

HiBench N N/A Two 10 3 N N N N

CALDA Y N/A One 5 1 Y N N N/YCSB Y N/A One 6 N/A Y N N NLinkBench Y One One 10 1 Y N N N

AMPBenchmarks

N N/A One 4 1 Y N N N

Observations from CloudSuiteObservations from CloudSuiteThe characteristic of big data workloadsThe characteristic of big data workloads

High L1I miss • Frontend inefficiencies

Low ILPC i ffi i i• Core inefficiencies

LLC is good but overprovision• Data access inefficiencies• Data‐access inefficiencies

Less off‐chip bandwidth and MLP• Bandwidth inefficienciesBandwidth inefficiencies

Clearing the Clouds, ASPLOS 2012 Best paper

System Behaviors of BigDataBenchSystem Behaviors of BigDataBenchCPU utilization I/O wait ratio

Diversified system level behaviors:

100%CPU utilization I/O wait ratio

-Wor H M

M-P H-

iffI-S

Wor S- S-O H I-O S S

… … …

Weighted disk I/O time ratio

S H I-S S A

High CPU utilization & less I/O time

-Wor H M

M-P H-

iffI-S

… … …

S H I-S S A

High CPU utilization & less I/O timeLow CPU utilization

Low CPU utilization relatively and lots of I/O time

-Wor H M

M-P H-

iffI-S

… … …

S H I-S S A

System Behaviors Of BigDataBenchSystem Behaviors Of BigDataBench CPU utilization I/O wait ratio

High CPU utilization & less I/O timeLow CPU utilization

Low CPU utilization relatively and lots of I/O timeM di CPU tili ti

-Wor H M

M-P H-

iffI-S

Medium CPU utilization and I/O

… … …

S H I-S S A

Workloads Classification Of BigDataBench

Finding from system behaviorsSystem behaviors vary across different workloads y yWorkloads can be divided into 3 categories:

IPC of BigDataBench vs. other

benchmarks

ay M-B

ordC M

ig TPAV

The average IPC of the big data workloads are larger than CloudSuite, SPECFP and SPECINT, same as PARSEC and slightly smaller than HPCCPARSEC and slightly smaller than HPCC

The avrerage IPC of BigDataBench is 1.3 times of that of CloudSuite

Some workloads have high IPC (M_Kmeans, S‐TPC‐DS‐Query8)

Instructions Mix of BigDataBench vs. other benchmarks

For big data workloads:b hMore branch instructions

The percentage is 20% (1.5 times larger than others), except for TPC‐CThe ratio of integer instructions to FP instructions is very high

The average is 73

Cache Behaviors of BigDataBench th b h kvs. other benchmarks

CPU‐intensive I/O‐intensive hybrid

L1I MPKIL1I MPKILarger than traditional benchmarks, but lower than that of CloudSuite (12 Vs. 31)

Different among big data workloadsDifferent among big data workloadsCPU‐intensive(8), I/O intensive(22), and hybrid workloads(9)

MPI workloads have less instruction cache miss

Only 3.4 on the average

Cache Behaviors of BigDataBenchCache Behaviors of BigDataBench

L2 Cache：The IO‐intensive workloads undergo more L2 MPKI

L3 Cache:The average L3 MPKI of the big data workloads is smallerThe average L3 MPKI of the big data workloads is smaller than all of the other workloads

The underlying software stacks impact dataThe underlying software stacks impact data locality

MPI workloads have better data locality and less cache misses

Our observation from BigDataBenchOur observation from BigDataBench

Unique characteristicMore branch instructions & Higher ratio of integer to FP instructions

Different behaviors between Big Data workloadsDisparity of ILP and memory access behaviorsDisparity of ILP and memory access behaviors• Several workloads can achieve higher IPC • Several workloads can achieve higher Off‐chip bandwidth

CloudSuite is a subclass of Big Data

High front‐end stall is not the unique characteristics of big data workloads

Related with software stacks

BigDataBench PublicationsBigDataBench Publicationsi h i h k S i f S i 20 hBigDataBench: a Big Data Benchmark Suite from Internet Services. 20th IEEE

International Symposium On High Performance Computer Architecture (HPCA‐2014).Characterizing and Subsetting Big Data Workloads. Zhen Jia, Jianfeng Zhan, Wang Lei, Rui Han, Sally A. McKee, Qiang Yang, Chunjie Luo, and Jingwei Li. In 2014 IEEE International Symposium on Workload Characterization (IISWC). IEEE, 2014.Characterizing data analysis workloads in data centers. 2013 IEEE International Symposium on Workload Characterization (IISWC 2013)（Best paper award）BigOP: generating comprehensive big data workloads as a benchmarking framework. 19th International Conference on Database Systems for Advanced Applications (DASFAA 2014)BDGS: A Scalable Big Data Generator Suite in Big Data Benchmarking. The Fourth workshop on big data benchmarking (WBDB 2014)

BigDataBench usersBigDataBench users

More than 20 groups have published papers using BigDataBenchg g

h // f i /Bi D B h/ /http://prof.ict.ac.cn/BigDataBench/users/

OutlineOutline

What is BigDataBench?

BigDataBench benchmarking methodology

Five StepsFive Steps

Multi‐tenancy

Big data

Diverse implementation

yor subset

Investigate

Typical workloads and data set

benchmark specification

important application domain

BigDataBench MethodologyBigDataBench Methodology

Application Benchmark Real‐world data sets

Multi‐tenancy

Domain 1

Application

Data models of different

types & semantics

specification 1

Benchmark Data generation

version

Mix with different percentages

Domain … Data operations &

workload patterns

specification …

Reduce benchmarking cost

Application

Domain N

Benchmark

specification N

Workloads with diverse implementations

BigDataBench

subset

benchmarking cost

Investigate Application DomainsInvestigate Application DomainsWh t li tiInternet Services What application domains should we pay attention to？

Internet Services

Internet ServicesInternet Services

Search Engine Social Network

Taking up 80% of internet services

Search Engine Social NetworkElectronic Commerce Media StreamingOthers

according to page views and daily visitors

http://www.alexa.com/topsites/global;0Top 20 websites

Multimedia DataMultimedia Data

The Explosive Growth of Multimedia Data

VIDEOS on YouTube

PHOTOS on FLICKR every hours MUSIC streaming every minute minute on PANDORA every minute

VIDEO feeds fromdata growth

arei t VOICE llVIDEO feeds from surveillance cameras

are IMAGES, VIDEOS, documents, …

minutes VOICE calls on Skype every minute

htt // ld l / t t/ l d /2014/11/ h ti bi d t DKB 2 df

http://www.oldcolony.us/wp‐content/uploads/2014/11/whatisbigdata‐DKB‐v2.pdf

The Explosive Growth of Human Genome Data

htt // h / it /d f lt/fil / i d bi d t 1 0 df

http://www.osehra.org/sites/default/files/genomics_and_big_data_1_0.pdf

Application Domain We ChooseApplication Domain We Choose

Internet ServicesServices

Search EngineSocial NetworkSocial NetworkE‐commerceMultimedia

Emerging

MultimediaBioinformatics

but Important

Application

Domain 1

Application

types & semantics

workload patterns

Application

Domain N

Success story: Relational model of data

E. F. Codd, A relational Model of Data for Large shared data banks Communication ofLarge shared data banks. Communication of ACM, vol 13. no.6, 1970.Set concept : general mathematical meaning

General representation of datapBasis of relational algebra (theoretical foundation of database)of database)5 basic operations

S l P j P d U i Diff• Select, Project, Product, Union, Difference

Success story: parallel computing

By a multidisciplinary group of well‐known researchers

e.g.:Jim Gray,Michael JordanDavid A. Patterson

Operations & PatternsAbstracted from 13representative parallelcomputation patternsParallel computation inherent demand for big data

BigDataBench MICRO 2014processing (volume & complexity)

Primitive Operations & Patterns in Big DataPrimitive Operations & Patterns in Big Data

3 Categories of Operations11 basic operations

3 P i P tt3 Processing Patterns

BigOP: generating comprehensive big data workloads as a benchmarking framework.

g g g p g gDASFAA 2014

Some Examples (not exhausted)Some Examples (not exhausted)

BigOP: generating comprehensive big data workloads as a benchmarking framework. g g g p g g19th International Conference on Database Systems for Advanced Applications (DASFAA 2014)

Image SearchImage Search

Perceptual Hash Algorithm

Input Image

samping Basic information

of image

SIFT Imagefeatures

Hash Imagefingerprint Database

Set Ope

SimilarSort

ration

SimilarimagesOutput

Feature Extraction SIFTFeature Extraction‐‐SIFT

GaussianFilter

InputImage Convolution

Image ScaleSpace

Sampling ImagePyramid

MatrixSubtraction DOG

Key PointOf Image

GaussianWindow

CountFeatureVectorsOutput Sampling

BigDataBench MICRO 2014 43

Workload & Data Set in BigDataBench

•Text•Graph•Table•Multimedia

• Structured• Semi‐Structured•Unstructured

Data Model Semantics

Workload Patterns

Data Operations

•Different combination of units of computation

•Unit of computation

Application Benchmark

Domain 1

Application

types & semantics

specification 1

Benchmark pp

workload patterns

specification …

Application

Domain N

Benchmark

specification N

Data management’s traditionData management s tradition

Specification First. Functions of abstraction are units ofFunctions of abstraction are units of computation that appear frequently in the application domain being benchmarkedapplication domain being benchmarked. They are expressed in a generic form that is independent of the underlying system implementation.implementation.

TPC C examplesTPC‐C examples

Domain 1

Application

types & semantics

specification 1

Benchmark Data generation pp

workload patterns

specification …

Application

Domain N

Benchmark

specification N

Real World Data setsReal‐World Data sets

14 real‐world data sets

Big Data Generation Tool BDGSBig Data Generation Tool‐‐BDGS

Provide scalable data set extracted from real‐world data sets

BDGSbased on real data

TableGraph

Naïve Text generatorNaïve Text generator

select word randomly

system

miningdata

evaluatemachine

select word randomlyarchitectureCPU

benchmarkingmemory

learning

wordsfollowing multinomial distribution

documnet

Only modeling on word level;Words are selected according to the same gdistribution

BigDataBench MICRO 201451/

Improved Text generatorImproved Text generator

big CPUevaluatemachine

topic2 CPUselect word randomly

architecture

benchmarking

miningdata

topic1

topic3

select topic randomlyCPU

systemmemorylearning

documnet

topic3

topicsfollowing multinomial distribution under topic2following multinomial distribution

Modeling on topic and word levelWords are drew from distribution under particular topicTopics are drew from same distribution, as a result, each document has same topic proportion

Optimized Text GeneratorOptimized Text Generator

big systemdata

evaluatemachine

topic1topic2

select word randomly

architecture

benchmarking

miningdata

memory

topic3

select topic randomlyselect topic

proportion randomly

following multinomial distribution under topic1

learning

documnettopics

following multinomial distribution

Topic distribution parameters

following dirichlet distribution

Modeling on topic and word levelWords are drew from distribution under particular topicTopics are selected from different distribution with parameters following a dirichlet distribution

Workloads With Diverse Implementations

MapReduce

Software Stack

DataMPIMPI

Multi‐tenancy

Domain 1

Application

types & semantics

specification 1

Benchmark Data generation

version

Mix with different percentages

workload patterns

specification …

Reduce benchmarking cost

Application

Domain N

Benchmark

specification N

BigDataBench

subset

benchmarking cost

Multi‐tenancy version of BigDataBench

Scenarios of multiple tenants running heterogeneous applications in cloud datacenters

Latency‐critical online servicesLatency‐insensitive offline batch applications

Mining real‐world Workload traces Mixed workloads

Benchmarking scenarios

Workload traces (Google and Facebook) Workload matching

using Parametric workload

Mixed workloads in public clouds

Profiling Real‐world Workload

Machine learning techniques

generation tool Data analytical

workloads in private clouds

traces

BigDataBench SubsetBigDataBench Subset

MotivationExpensive to run all the benchmarks for system p yand architecture researches

• multiplied by different implementationsmultiplied by different implementations • BigDataBench 3.0 provides about 77 workloads

Subsetting MethodologySubsetting Methodology

• Identify a comprehensive set of workload characteristics from a specific perspective

• Eliminate the correlation data in those metricsEliminate the correlation data in those metrics• Map the high dimension metrics to a low dimension

• Use the clustering method to classifyCh t ti kl d f h t• Choose representative workloads from each category

BigDataBench First Part-1213 - Big Data and AI Benchmark...

Documents

INSTITUTE OF COMPUTING TECHNOLOGY Understanding Big Data Workloads on Modern Processors using BigDataBench Jianfeng Zhan

AVAILABLE FOR SALE/LEASE - LoopNet...413,415,417 Northland Blvd -ONE SALE for all 3 Bdgs FOR LEASE INDIVIDUALLY AVAILABLE FOR SALE/LEASE FEATURES: • 413- 7,680 Total SF • 415-

BigDataBench: A Scalable and Uniﬁed Big Data and AI ...micro-architectural performance of AI benchmarks on GPUs. Index Terms—Big data, AI, data motif, uniﬁed benchmarks, scalability,

GOVERNMENT OF BRUNEI DARUSSALAM ...mfa.gov.bn/PublishingImages/Pages/BDScholarship/BDGS...to my acceptance of the Government of Brunei Darussalam Scholarship for Foreign Students Scholarship

BDGS: A Scalable Big Data Generator Suite in Big Data ... · BDGS: A Scalable Big Data Generator Suite in Big Data Benchmarking 3 Speci cally, our BDGS can generate data using four

Benchmarking Datacenter and Big Data Systemsprof.ict.ac.cn/BigDataBench/old/2.0/BigDataBenchmarking_xi'an.pdf · Big Data Benchmarking Workshop DCBench DCBench: typical data center

BigDataBench - ict.ac.cnprof.ict.ac.cn/DComputing/uploads/2013/DC_4_2_BigData... · 2013. 7. 10. · INSTITUTE OF COMPUTING TECHNOLOGY BigDataBench: a Big Data Benchmark Suite from

BigDataBench Technical Report - ict.ac.cnprof.ict.ac.cn/BigDataBench/wp-content/uploads/... · analytics workloads using MapReduce, MPI, Spark, DataMPI, interactive an-alytics and

BigDataBench : a Big Data Benchmark Suite from Internet Services

BigDataBench - A Big Data Benchmark Suite From Internet Services

Handbook of BigDataBench (Version 3.1)|A Big Data Benchmark …prof.ict.ac.cn/wp-content/uploads/2014/12/BigDataBench... · 2017-06-12 · Handbook of BigDataBench (Version 3.1)|A

Mbench: Benchmarking a Multicore Operating System Using ...prof.ict.ac.cn/bpoe_6_vldb/wp-content/uploads/2015/... · Multicore Operating System Using Mixed Workloads ... The seven

Exploring Opportunities for Non-Volatile Memories in Big ...prof.ict.ac.cn/bpoe_4_asplos/wp-content/uploads/... · examine the implications of big data workloads on system design

Characterizing and Benchmarking Deep Learning Systems on ...prof.ict.ac.cn/Bench18/luxi-bench18-final.pdf · • AR-gRPC incurs a maximum speedup of 3x(55 vs 18 images) compared to

Workload-Aware Aggregate Maintenance in Columnar In-Memory Databases - Big Data …prof.ict.ac.cn/bpoe2013/downloads/papers/BigD389_5597.pdf · 2013-09-13 · Workload-Aware Aggregate

INSTITUTE OF COMPUTING TECHNOLOGY BigDataBench: a Big Data Benchmark Suite from Internet Services Lei Wang, Jianfeng Zhan, Chunjie Luo, Yuqing Zhu, Qiang

BigDataBench: a Big Data Benchmark Suite from Internet ...arXiv:1401.1406v2 [cs.DB] 22 Feb 2014 BigDataBench: a Big Data Benchmark Suite from Internet Services Lei Wang1,7, Jianfeng

PLC-Cache: Endurable SSD Cache for Deduplication-based ...prof.ict.ac.cn/wp-content/uploads/2014/04/msst... · primary and secondary deduplication. Primary data deduplication is implemented

THE DATACENTER NEEDS AN OPERATING SYSTEM - Big Data …prof.ict.ac.cn/DComputing/uploads/2012/DC_4_2_hotcloud_datacent… · THE NEW COMPUTER Running today’s most popular consumer

prof.ict.ac.cnprof.ict.ac.cn/BigDataBench/wp-content/uploads/... · Web viewAlso, the basic operations like sort, word count, and grep should be contained. To covering the Hive and