MapR Enterprise Data Hub Webinar w/ Mike Ferguson

®© 2014 MapR Technologies 1

®

© 2014 MapR Technologies

Best Practices for Using Hadoop as an Enterprise Data Hub Mike Ferguson – Intelligent Business Strategies Steve Wooledge – MapR June 18, 2014

2

About Mike Ferguson

Mike Ferguson is Managing Director of Intelligent Business Strategies Limited. As an analyst and consultant he specialises in business intelligence, data management and enterprise business integration. With over 32 years of IT experience, Mike has consulted for dozens of companies, spoken at events all over the world and written numerous articles. Formerly he was a principal and co-founder of Codd and Date Europe Limited – the inventors of the Relational Model, a Chief Architect at Teradata on the Teradata DBMS and European Managing Director of DataBase Associates.

www.intelligentbusiness.biz [email protected]

Twitter: @mikeferguson1

Tel/Fax (+44)1625 520700

The Hadoop Data Refinery and Enterprise Data Hub

Mike Ferguson Managing Director Intelligent Business Strategies June 2014

4

Topics

!  Data warehousing and the evolution of ETL processing

!  New data and new analytical workloads

!  Big data use cases driving business agendas

!  The unprecedented demand for customer insight

!  Challenges with new big data sources

!  Beyond the data warehouse – new platforms for new analytical workloads

!  The role of Hadoop in the modern analytical ecosystem

!  Introducing the Hadoop enterprise data hub and data refinery

!  Simplifying access to new big data insight using SQL on Hadoop

!  Integrating Hadoop into your analytical ecosystem

5

For Many Years The Traditional Data Warehouse and BI Environment Has Been Used For Analysis & Reporting

Operational systems

web

P o r t a l

Employees Partners

Customers

BI Tools

Platform Dat

a In

tegr

atio

n / D

Q

Reports & analytics

Data warehouse & data marts

DW

6

The Evolution of Data Integration in Data Warehousing – From Hand Coded to ETL to ELT

Hand coded ETL programs

DW Hand coded

programs

ETL Servers

DW ETL

Servers

ELT processing

Generated SQL ELT

processing

DW Evolution of Data Warehousing

MPP RDBMS systems

7

Sales

Product line n

Product line 4

Product line 3

Product line 2

Product/service line 1

Marketing

Service

Credit Verification

HR

Finance

Planning

Procurement

Sup

ply

Cha

in

Sup

plie

rs

Front Office BackOffice

Operations

Cus

tom

ers

New Data Sources Have Emerged Inside And Outside The Enterprise That Business Now Wants To Analyse

E.g. RFID tag

sensor networks

weather data Data volume Data variety Number of sources

Data volume Data velocity

8

Popular Big Data Analytic Applications – Web Data

!  Clickstream analytics •  Site navigation behaviour (session) analysis

–  Paths to buy, paths to abandonment, what else they looked at

–  Improve customer experience and conversion –  Associate clicks with customers & prospects

!  Social network influencer analysis •  Graph analytics for influencer behavioural impact

analysis •  ‘Target the influencer’ marketing campaign

effectiveness

9

Popular Big Data Analytic Applications – Sensor Data For Improving Process Efficiency and Optimisation

!  Sustainability analytics e.g. energy optimisation !  Supply/distribution chain optimisation !  Asset management and field service optimisation ! Manufacturing production line optimisation !  Location based advertising (mobile phones) ! Grid health monitoring

•  Electricity, water, mobile phone cell network…

!  Smart metering (collect data every 15 minutes) !  Fraud !  Healthcare – ITC vital signs, fit bits,…. !  Traffic optimisation " WHAT ARE YOU PREPARED TO INSTRUMENT?

E.g. RFID tag

10

Popular Big Data Analytic Applications – Unstructured Data

!  Case management

!  Fault management and field service optimisation

!  “Voice of the customer”

!  Sentiment analytics

!  Competitor analysis

! Media coverage analysis

!  Improve pharma drug trials

" Unstructured content is hard to analyse

How much is TEXT worth to your business?

11

Big Data Analytics - Industry Use Case Examples

Industry Use Case Examples Financial Services

Improved risk decisions, KYC customer insight, auto programmatic trading, 360 view of financial crime, pre-trade decision support, real-time trade & corp action tagging for compliance and RT P&L, grow security services outsourcing, Reference Data Exchange

Utilities Smart meter data analysis, pricing elasticity analysis, customer loyalty, sustainability, asset management

Telecommunications

Customer Churn, Network optimization analysis from device, sensor and GPS inputs, monetization of GPS and data

Manufacturing Sensor data for next generation ‘smart’ products, production line optimisation, improved customer service and improved field service, distribution chain optimization, asset management

Insurance “How you drive” insurance (sensors to reduce risk), broker document analysis (risk assessment)

Government Smart cities (e.g. transportation optimisation), anti-terrorism, law enforcement

Logistics Distribution optimisation, route optimisation,

12

More Data Is Required To Get A Deeper Understanding of Customers

! We now need •  Transaction data •  Data from touch points you own •  Data from the touch points you don’t own •  Interaction data

–  Need to look at Inbound interactions Vs outbound interactions –  Social interactions

•  Master data •  Professional data e.g. profiles on LinkedIn •  Internal and external event data •  Competition data…..

!  Then use analytics to understand and predictive desire and propensity e.g. propensity to churn

13

Top Priorities - Improving Customer Experience Via Time Series Analysis of All Customer Interactions

OMNI channel – analyse all customer interactions across all channels

identity data

behavioural data

social data

Customer “DNA”

14 identity

data behavioural data

social data

Customer “DNA”

Customer Experience Management - Understanding Customer On-Line Behaviour is Mission Critical to Retention and Growth

!  Important new data sources for analysis for customer ‘DNA’ •  Clickstream data from web logs •  Sentiment and social network influencer data

New competitors

More choice

Voice of the customer

On the web the customer is king

On the move

Easy to find

15

Today Both Structured And Multi-Structured Data Are Needed For Deeper Insight

Multi-structured

data Click stream web log data Customer interaction data

Social interaction data Sensor data

Rich media data (video, audio) External content

Documents Internal web content

Seismic data (oil & gas)

Structured data

OLTP system data Data warehouse data

Personal data stores e.g. Excel, Access

Often un-modelled and may not be well understood

Often a schema is defined and data is well understood

Data characteristics are changing - Companies must deal with volume,

variety and velocity

16

Big Data Analytics Challenges Include The Analysis of Unstructured, Semi-structured and Structured Data

{ "firstName": ”Wayne", "lastName": ”Rooney", "age": 25, "address": { "streetAddress": "21 Sir Matt Busby Way", "city": ”Manchester”, “country”: “England”, "postalCode": “M1 6DY” }, "phoneNumbers": [ { "type": "home”, "number": ”0161-123-1234” }, { "type": ”mobile", "number": ”07779-123234” } ] } JSON data

Text data

Image Data

Makes analysis more complex with new analytics and visualisations needed

17

Increased Data and Analytical Complexity Has Created A Need For A New Role – The Data Scientist

Image source: Wikipedia

Data Science is the process of investigative / exploratory analysis of multi-structured data to discover and produce new business insights

Image source: www.computing.co.uk

18

People In Different Roles In The Analytical Landscape Need To Work Together To Deliver Business Value

Exploratory analysis Predictive / statistical model producer

Business Analyst

Business Manager / Operations worker /

Customer Data Scientist

Model consumer Data visualisation Information Producer

• Build reports • Build and publish dashboards

Information consumer Decision maker Action taker

Strategic Business Objective

Priority KPI

Current KPI

Value

What is +1%

worth?

KPI Target

Executive Accountable

Business Initiatives (projects)

Budget Allocation

Action Plan

1 $$$ Project Project Project

£ x Million

2

3

4

Business Strategy – strategic objectives and targets including sustainability targets

sandbox

19

Data Science Produces New Insights For Business Analysts Who Produce Actionable BI For Front Office Decision Makers

Business Analyst Marketing Manager / Marketing, Sales and

Service workers Data Scientist

Data Quality

Forecasting

Segmentation

Models

Customer Lifetime Value

Social Network

Strategy Creation

Performance & Effectiveness

Reporting

Direct Mail

Understand Customer Behavior

& Navigation

Marketing Performance &

Reporting

Campaign Planning

Financial Planning

Creative Materials

Marketing Attribution

Operations Management

Channel Efficiency

Sentiment & Influence

Dynamic Content

Re-marketing

Web

Call Center

Live Event

Broadcast Media

Mobile/ SMS

Social

Email

Industry Specific

Big Data Analytics Traditional DW/BI

Workflow & Approvals

New insights Actionable BI

20

Big Data Analytics Has Taken Us Beyond The Traditional DW – New Big Data Analytical Workloads

1.  Analysis of data in motion

2.  Complex analysis of structured data

3.  Exploratory analysis of un-modeled multi-structured data

4.  Graph analysis e.g. social networks

5.  Accelerating ETL and analytical processing of un-modeled data to enrich data in a data warehouse or analytical appliance

6.  The storage and re-processing of archived data

21

The Changing Landscape – We Now Have Different Platforms Optimised For Different Analytical Workloads Big Data workloads result in multiple platforms now being needed for analytical processing

Streaming data

Hadoop data store

Data Warehouse RDBMS

NoSQL DBMS

EDW

DW & marts

NoSQL DB e.g. graph DB

Advanced Analytic (multi-structured data)

mart DW

Appliance

Advanced Analytics (structured data)

Analytical RDBMS

Graph analysis

Investigative analysis,

Data refinery

Traditional query,

reporting & analysis

Real-time stream

processing & decision m’gmt

Data mining, model

development

22

Hadoop Is A Key Platform In Big Data Analytics – Data Can Be Accessed Via Multiple APIs

Java MapReduce APIs to HDFS, HBase, Cascading

file file file file file


file file

file file

webHDFS (An HTTP interface to HDFS has

REST APIs) HDFS

file

file

file

file

YARN

PIG latin scripts

SQL

Vendor SQL on Hadoop engine

MapReduce Application

index index Index partition

SQL

BI Tools & Applications

Storm

Application

YARN

Tez or Spark MapReduce HBase

HDFS API

23

Defacto Standard APIs Allow Hadoop Components To Be Replaced e.g. Faster, More Secure File System Than HDFS

Java MapReduce APIs to HDFS, HBase, Cascading

webHDFS (An HTTP interface to HDFS has

REST APIs) file file file file file


file file

file file

file

file

file

file Vendor Specific File System (e.g. )

YARN

HDFS API

PIG latin scripts

index index Index partition

Storm

Application

YARN

MapReduce HBase

MapReduce Application

SQL

Vendor SQL on Hadoop engine

SQL

BI Tools & Applications

Tez or Spark

24

Apache Hadoop Components Component Description

Hadoop HDFS A distributed file system that partitions files across multiple machines for high-throughput access to application data – HDFS API allows vendors to replace HDFS with an alternative

Hadoop YARN" A framework for job scheduling and cluster resource management"Hadoop MapReduce

A programming framework for distributed batch processing of large data sets distributed across multiple servers

Avro A serialization system that creates & reads files in a format containing both JSON data definitions & the data itself for dynamic interpretation of the data by applications

Hive A data warehouse system for Hadoop that facilitates data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop-compatible file systems. Hive provides a mechanism to project structure onto this data and query it using a SQL-like language called HiveQL. HiveQL programs are converted into MapReduce programs

HBase HBase is an open-source, distributed, versioned, column-oriented store modeled after Google' Bigtable.

Pig A high-level data-flow language for expressing Map/Reduce programs for processing and analysing large HDFS distributed data sets

Mahout A scalable machine learning and data mining library

Oozie A service for running and scheduling workflows of Hadoop jobs (including Map-Reduce, Pig, Hive, and Sqoop jobs)

Spark A general purpose engine for large scale data processing in-memory. It supports analytical applications that wish to make use of stream processing, SQL access to columnar data and analytics on distributed in-memory data

Zookeeper A high-performance coordination service for distributed applications

25

The Role of Hadoop - Data Is Arriving Faster Than We Can Consume It – How Good Is Your Filter?

F D I A L T T A E R

Enterprise

Enterprise systems

26

New Requirement – The Managed Hadoop Enterprise Data Hub

Parse & Prepare Data in Hadoop (MapReduce)

Transform & Cleanse Data in Hadoop (MapReduce)

Discover data in Hadoop

ELT work -flow

sandbox

other data

sandbox sandbox

Data Reservoir (raw data)

Load data into Hadoop

Data Refinery

New high value Insights

(pub/sub)

EDW Graph DBMS

DW appliance contains clean,

high value data

XML,%JSON%Web

logs

27

What’s In An Enterprise Data Hub?

!  A managed data reservoir (raw data) •  Organised capture of multi-structured data •  Includes real-time data capture •  May include operational reporting

!  A governed data refinery •  Data integration and cleansing at scale •  Analytical sandboxes to discover high value data

!  Published, protected and secure high value insights

!  Long-term storage of archived data from data warehouses

28

file file file

file file file

file file

file file

file

file

Real-time Data Capture – E.g. MapR Allows Web Log Data To Be Directly Streamed/Stored in Hadoop

MapR Direct Access NFSs allows Web log files to be stored directly on

their Hadoop File System so that click stream is captured in real-time

MapR Distribution for Hadoop

Web Server

Direct Access NFS

web log file web log

file

# mount localhost:/mapr /mapr

HDFS

Web Server Web Server

29

High Volume Data Capture - Column Family Databases !  Suitable for fast capture of large amounts of sparse, volatile data

•  Very fast capture and can hold vast amounts of data •  Billions of rows containing thousands or millions of columns

!  Provide column-centric storage and wide de-normalised big tables can also help simplify operational reporting if used with SQL-on-Hadoop e.g. SQL access to HBase

!  Allow you to •  Group together related columns into column families •  Design column families to optimize the most common queries •  Retrieve columnar data for multiple entities by iterating through a

column family •  Shard rows in a column family and distribute across many servers •  Create indexes and secondary indexes •  Support schema variance - columns in a column family can vary for

every row

30

NoSQL Column Family Databases - HBase

Row 1 # Column A = value Column B = value Column C = value

Row 2 # Column X = value Column Y = value Column Z = value

Hbase Storage Architecture

Hmaster and several HRegionServers

Regions (partitions) created automatically as tables grow Hbase allows applications to directly read and write data

31

Column Families Can Be Stored In Different Files And Queries Will Only Retrieve The Column Family Needed

Source: Data Access for Highly-Scalable Solutions : Using SQL, NoSQL, and Polyglot Persistence, McMurtry, Oakley, Sharp, Subramanian, Zhang

Portfolio.* means all columns in the Portfolio column family

Data about a customer and their stock purchases are partitioned vertically by column family

Column family data can also be compressed

32

Fast Data Capture – MapR-DB Is A High Speed Version of HBase Built Into The MapR Data Platform

HBase API

Source: MapR

33

Enterprise Data Hub – We Need A Data Refinery To Process And Clean Complex Data

Image source: http://www.hollyfrontier.com/navajo/

34

Evolution of Big Data Integration Is Following The Same Cycle as it Did in Data Warehousing

Hand coded ETL programs

Hadoop Hand coded

programs

ETL Servers

Hadoop ETL

Servers

ELT processing

Generated MapReduce ELT

processing

Hadoop Evolution of Big Data Integration

35

Data Cleansing and Integration Tool

Scaling ETL In A Data Refinery By Generating Pig, Hive or 3GL MapReduce Code for In-Hadoop ELT Processing

Extract Parse Clean Transform Analyse Load Insights

Option 1 ETL tool generates HQL or convert generated SQL to HQL

Option 2 ETL tool generates Pig Latin (compiler converts every transform to a map reduce job)

Note - Generating native MapReduce code instead of HiveQL or Pig Latin would likely perform faster because there is no need to translate into MapReduce Also HiveQL is a subset of SQL so check how ETL tools generating HiveQL do complex transformations – HiveQL on its own may not be enough e.g. Hive UDFs?

Option 3 ETL tool generates 3GL MapReduce code

36

Need to Parse & Extract From Multi-Structured Data While Integrating Data In A Big Data Environment

E-mail (semi-structured)

Text (unstructured)

Extract Parse Transform Load …

37

Sandboxes In The Data Refinery - Data Science Teams Need To Conduct Exploratory Analysis on Multi-Structured Data

Click stream web log data Customer interaction data

Social interaction data (e.g. Twitter, Facebook)

Sensor data Rich media data (video, audio)

External web content Documents

Internal web content Seismic data (oil & gas)

Investigative / Exploratory Analysis

C

R U

D

Asset Customer

Product

MDM System

EDW mart

new business insights

sandbox

Multi-structured data

Historical Data

archived DW data master data

Data Scientists

38

In-Hadoop Analytics In A Data Refinery – Example Technologies

!  Hadoop MapReduce, Tez or Spark analytic applications with custom analytics •  Pig, Java, Python, Scala, Cascading…..

!  Hadoop MapReduce, Tez or Spark analytic applications using pre-built Hadoop analytics e.g. Mahout, Spark MLlib •  Several analytical algorithms for use in analysis

!  Revolution Analytics RevoScaleR

!  SAS Analytics and In-Memory Statistics for Hadoop

!  … many more

Analytical tools

Data management

tools

39

In-Hadoop Analytics: - Mahout Supports A Number Of Analytic Techniques

!  Collaborative Filtering

!  User and Item based recommenders

!  K-Means and Fuzzy K-Means clustering

! Mean Shift clustering

!  Dirichlet process clustering

!  Latent Dirichlet Allocation

!  Singular value decomposition

!  Parallel Frequent Pattern mining

!  Complementary Naive Bayes classifier

!  Random forest decision tree based classifier

https://cwiki.apache.org/confluence/display/MAHOUT/Algorithms

Now runs on Spark as

well as MapReduce

40

Expediting The Data Refinery Process On Hadoop With Automated Analysis – From ETL to Analytical Workflows

Parse & Prepare Data in Hadoop (MapReduce)

Transform & Cleanse Data in Hadoop (MapReduce)

Discover data in Hadoop

ELT work -flow

other data

Raw data

Load data into Hadoop

Data Refinery

EDW Graph DBMS

DW appliance

Automated Invocation of Custom Built & Pre-built Analytics on Hadoop

contains clean, high value data

New high value Insights

(pub/sub)

41

High Value Insights Produced In A Hadoop Data Hub Can Be Brought Into A DW to Enrich What We Already Know

Cloud Data

HDFS

Extract

DW D I Map/ Reduce data

transformation and analytics applications

Transform

e.g. PIG, IBM JAQL

Cloud Data e.g. Deriving insight from huge volumes of social web content on sites like twitter, facebook. Digg, mySpace, tripAdvisor, Linkedin….for sentiment analytics

Hundreds of terabytes up to petabytes

new insights

Operational systems

42

Making New Insights Available To Business Analysts Via SQL Access To Big Data - Options

SQL

SQL access to big data in Hadoop

SQL

DW

data virtualisation server

SQL access to big data via data

virtualisation

SQL

Analytical RDBMS

SQL access to big data in an

analytical RDBMS

streaming data

SQL

SQL access to streaming data in

motion

43

Self-Service BI

BI Tool(s) e.g, Visual Discovery tools

Business Analyst or ‘budding’ Data

Scientist

personal & office data

Predictive models

community

Publish / Share Consume / Enhance / Re-publish

Transaction systems

DW

SQL Access to Hadoop Is Needed To Allow Hadoop Data To Be Accessed By Users With Self-Service BI Tools

collaborate

HDFS / Hbase/ Hive

e.g. Hive interface

44

SQL access to Big Data?

Key Questions That May Influence If SQL Access to Big Data Is A Good Choice or What SQL Option to Take

What kind of analysis? Text analysis, Graph analysis, Machine Learning, reporting

What kind of data type(s) do you need to analyse? - structured, unstructured, semi-

structured,

What kind of data volumes do you want to analyse?

Is the data at rest or is it real-time streaming data in motion?

What analytical functions can you invoke on big

data from SQL?

Join with other data in another data store?

How many concurrent users?

Performance and scalability of complex queries and

analytical functions (need parallelism)

Is the requirement for interactive, exploratory, or real-time analysis?

Data

Analytical Workload

45

SQL On Hadoop Initiatives

Key Questions What analytic functions are provided? How can analytic functions be extended Can you join to data outside of Hadoop? Are these SQL on Hadoop options suitable for reporting and analysis, interactive discovery, exploratory analysis or all of these?

Vendor SQL on Hadoop Initiative AMPlab (UC Berkeley) Shark (Forked Hive at V0.9) or SparkSQL

Apache Hadoop Hive

Actian Vortex (Actian Vector on Hadoop data nodes)

CitusDB CitusDB (uses external tables)

Cloudera Impala / Parquet

Concurrent Lingual (SQL on Cascading)

Hadapt Schemaless SQL

Hortonworks Stinger / ORC (Hive 13)

HP Vertica on Hadoop

IBM BigSQL (SQL on HDFS & HBase)

InfiniDB InfiniDB on Apache Hadoop

Jethro Data JethroData

MapR Apache Drill

Microsoft Hive 13

Pivotal HawQ (uses external tables via PFX)

Teradata SQL-H

Splice Machine Splice Machine (SQL Engine on HBase)

Salesforce.com Phoenix (SQL engine on HBase)

Attivio Active Intelligence Engine (SQL access to search indexes on Hadoop data)

46

SQL on Hadoop – Apache Drill Can Access HDFS And HBase Data

BI Tool(s) e.g, Visual Discovery tools

Business Analyst or’ Data Scientist

Drill

Analytic Application

SQL SQL

Data Scientist

HDFS HBase

MapR Distribution for Hadoop

Apache Drill does not use MapReduce

MongoDB/ Cassandra

sensors

XML,%JSON%

Data entering HBase

47

Apache Drill Distributed Query Processing – A Storage Independent Drillbit MPP Architecture

Each drillbit is capable of receiving queries from applications and BI tools - there is no master in this architecture Multiple drillbits are involved in parallel query processing on distributed data

Supports Apache HDFS, Apache HBase, MapR-FS, MapR-DB, Amazon S3

48

SQL on Hadoop Example – Apache Drill Supports Query of Self-Describing Data Without a Schema

JSON

Source: MapR

49

file

file

file

file

file

file file

file

file

file

file

SQL on Hadoop – What Should The Schema Look Like?

Star schema? Snowflake schema?

De-normalised schema?

Other?

50

Hadoop Storage Is Independent of Any SQL Engine Accessing HDFS - Multiple SQL Engines Can Coexist On The Same Data



file file

file file

HDFS file

file

file

file

YARN

Batch (MapReduce)

Interactive (Tez)

On-line (HBase)

Streaming (Storm,..)

Graph (Giraph)

In-memory (Spark)

HPC MPI (OpenMPI)

Other (Search,.)

file

file

file

file

SQL SQL SQL SQL

Storage is independent of any SQL engine !  Key points about Hadoop

•  It is possible to have MULTIPLE SQL engines on the same data •  Different SQL engines run on different Hadoop frameworks (M/R, Tez,

Spark) or on no framework at all i.e. directly access HDFS or HBase data

51

Relational DBMS / Hadoop Integration – Several Vendors Have Integrated RDBMS with Hadoop to Run Analytics

Relational DBMS

External Polymorphic

table function(s)

HDFS / Hbase/ Hive

SQL, XQuery

RDBMS optimizer handles transparent access to external analytical platforms on behalf of the user

RDBMS and Hadoop could be deployed on the same hardware cluster (preferred) or on different hardware clusters

Allows join across data in a single RDBMS and Hadoop

52

Relational DBMS / Hadoop Integration Example - HP Vertica and MapR

Source: MapR

53

Self-Service BI

Self-service Data Discovery & Visualisation

or Dashboard Server

Business analyst

Data Virtualization and Optimization

personal & office

data Predictive models

Transaction systems

Data Management Tools (ETL, DQ, etc.)

DW

Self-Service Access To Big Data Via Data Virtualization

BUT what about optimization? Can the data virtualisation server push down analytics to underlying platforms to make them do the work?

54

New Insights Can Be Added Into A Data Warehouse To Enrich What You Already Know

DW D I

new insights

Operational systems

e.g. Deriving insight from social web sites like for sentiment analytics

sandbox

Data Scientists

social

Web logs

web cloud ELT

55

Alternatively New Insights In Hadoop Can Integrated With A DW Using Data Virtualization To Provide Enriched Information

DW D I

e.g. Deriving insight from social web sites like for sentiment analytics

new insights

OLTP systems

sandbox

Data Scientists

social

Web logs

web cloud

Data Vitualisation

SQL on Hadoop

56

Using Hadoop As A Data Archive Means Data Can Be Kept On-line, Analysed And Still Integrated With Data In The DW

DW D I

OLTP systems

Data Vitualisation

SQL on Hadoop

Archived data

Archive unused

or data > n years

57

SQL on Hadoop

Big Data Governance – Data Sources, Sandboxes, People, Data Access Security, Results Lineage….

Graph DBMS

MPP Analytical RDBMS

Social graph data Unstructured / semi-

structured content

DW

RDBMS Files clickstream%

Web logs

governance

governance

governance

governance

governance

governance

governance governance governance

58

Issues: Siloed Analytics - Different Tools to Manage and Integrate Data For Each Type of Analytical Data Store

Analytical tools

Data management

tools

EDW mart

Structured data

CRM ERP SCM

Silo

DW & marts

Streaming data (markets, sensors

Analytical models

Silo

Analytical tools/apps

Data management

tools

Multi-structured data

Silo

DW Appliance


Data management

tools

Structured data

CRM ERP SCM

Analytical tools

Silo

Analytical tools/apps

Data management

tools


Silo

Multi-structured & structured data

59

EDW

MDM System DW & marts


Advanced Analytic (multi-structured data)

mart DW

Appliance


Need to Manage The Supply of Consistent Data Across The Entire Analytical Ecosystem

Common Enterprise Information Management Tool Suite Stream

processing

C

R

U

D

Prod

Asset

Cust

actions

feeds sensors

XML,%JSON%

RDBMS Files office docs social Cloud clickstream%

Web logs web services

New

New

New

New

New New New New New New

New

New

C

R

U

D

Prod

Asset

Cust

New data types need to be supported by EIM tool suites

60

BI tools platform & data visualisation

tools

Search based

BI tools

Custom MapReduce applications

Map Reduce BI tools

Graph Analytics

tools

A New Architecture for Analytics - The Intelligent Business Strategies Extended Analytical Ecosystem

Enterprise Information Management Tool Suite

feeds sensors

XML,%JSON%

RDBMS Files office docs social Cloud clickstream%Web logs web services

Event processing

C

R

U

D

Prod

Asset

Cust

EDW

MDM System DW & marts


Advanced Analytics (multi-structured data)

mart DW

Appliance


actions

Filtered

data

Data Virtualisation and optimization

61

Conclusions

!  Business demand for new more complex, high volume data is driving the need for new analytical workloads beyond the data warehouse

!  Hadoop is a low cost analytical platform capable of supporting new analytical workloads on multi-stuctured data

!  A key role for Hadoop is as an data hub and data refinery

!  The data refinery process requires data integration and cleansing to scale to handle the volume, variety and velocity of complex multi-structured data

!  Data scientists analyse big data as part of the data refining process to produce new insights that can be added to what you already know

!  Hadoop is part of an extended analytical ecosystem with data management tools supplying consistent data across all data stores

!  Data scientists, business analysts and information consumers need to work together to deliver new insight for competitive advantage

®© 2014 MapR Technologies 62 © 2014 MapR Technologies ®

Best Practices for Production Success


HQ

WORLDWIDE HADOOP TECHNOLOGY LEADER UNIQUELY ADDRESSES BOTH ANALYTIC AND OPERATIONAL USE CASES 500+ PAYING CUSTOMERS

MapR:


MapR: Best Product for Customer Success

Top Ranked Exponential Growth 500+ Customers

3X bookings Q1 ‘13 – Q1 ‘14

80% of accounts expand 3X

90% software licenses

< 1% lifetime churn

> $1B in incremental revenue generated by 1 customer


FOUNDATION

Architecture Matters for Success


FOUNDATION

High Availability & Data Protection

High performance

Multi-tenancy

Operational & analytical workloads

Open standards for integration

NEW APPLICATIONS SLAs TRUSTED INFORMATION LOWER TCO

Architecture Matters for Success


The Power of the Open Source Community M

anag

emen

t

MapR Data Platform

APACHE HADOOP AND OSS ECOSYSTEM

Security

YARN

Pig

Cascading

Spark

Batch

Spark Streaming

Storm*

Streaming

HBase

Solr

NoSQL & Search

Juju

Provisioning &

coordination

Savannah*

Mahout

MLLib

ML, Graph

GraphX

MapReduce v1 & v2

EXECUTION ENGINES DATA GOVERNANCE AND OPERATIONS

Workflow & Data

Governance

Tez*

Accumulo*

Hive

Impala

Shark

Drill*

SQL

Sentry* Oozie ZooKeeper Sqoop

Knox* Whirr Falcon* Flume

Data Integration & Access

HttpFS

Hue

*%Cer6fica6on/support%planned%for%2014%


MapR Distribution for Hadoop M

anag

emen

t

MapR Data Platform

APACHE HADOOP AND OSS ECOSYSTEM

Security

YARN

Pig

Cascading

Spark

Batch

Spark Streaming

Storm*

Streaming

HBase

Solr

NoSQL & Search

Juju

Provisioning &

coordination

Savannah*

Mahout

MLLib

ML, Graph

GraphX

MapReduce v1 & v2

EXECUTION ENGINES DATA GOVERNANCE AND OPERATIONS

Workflow & Data

Governance

Tez*

Accumulo*

Hive

Impala

Shark

Drill*

SQL

Sentry* Oozie ZooKeeper Sqoop

Knox* Whirr Falcon* Flume

Data Integration & Access

HttpFS

Hue

*%Cer6fica6on/support%planned%for%2014%

•  High availability •  Data protection •  Disaster recovery

•  Standard file access

•  Standard database access

•  Pluggable services •  Broad developer

support

•  Enterprise security authorization

• Wire-level authentication

•  Data governance

•  Ability to support predictive analytics, real-time database operations, and support high arrival rate data

•  Ability to logically divide a cluster to support different use cases, job types, user groups, and administrators

•  2X to 7X higher performance

•  Consistent, low latency

Enterprise-grade Security Operational Performance Multi-tenancy Interoperability


Hadoop + Data Warehouse Architecture Improve data services to customers without increasing enterprise architecture costs

•  Provide cloud, security, managed services, data center, & comms •  Report on customer usage, profiles, billing, and sales metrics •  Improve service: Measure service quality and repair metrics

•  Reduce customer churn – identify and address IP network hotspots •  Cost of ETL & DW storage for growing IP and clickstream data; >3

months •  Reliability & cost of Hadoop alternatives limited ETL & storage offload

•  MapR for data staging, ETL, and storage at 1/10th the cost •  MapR provided smallest datacenter footprint with best DR solution •  Enterprise-grade: NFS file management, consistent snapshots & mirroring •  Data warehouse for mission-critical reporting and analysis

OBJECTIVES

CHALLENGES

SOLUTION

Hadoop + Data Warehouse = New, Deeper Insights for the Business •  Increased scale to handle network IP and clickstream data •  Freed up processing on DW to maintain reporting SLA’s to business •  Unlocked new insights into network usage and customer preferences

Business Impact

FORTUNE 500 TELCO


Q & A Engage with us!

@mikeferguson1 – Intelligent Business Strategies @swooledge – MapR Technologies

•  Learn more about Hadoop in your architecture: www.mapr.com/EDH

•  Upcoming Webinar series - www.mapr.com/resources/webinars –  6/26 Talend – ETL in/for Hadoop –  7/09 Syncsort – comScore & mainframe optimization –  7/17 Rick van der Lans – SQL-on-Hadoop –  7/23 Skytree – machine learning & analytics –  7/30 Appfluent – DW usage monitoring & optimization –  8/14 Tableau – data exploration & analysis on Hadoop

•  Contact / follow us

Technology

MapR Enterprise Data Hub Webinar w/ Mike Ferguson