BI congres 2016-2: Diving into weblog data with SAS on Hadoop - Lisa Truyers - Keyrus

BIG DATA ANALYTICS

BUSINESS INTELLIGENCE

INFORMATION MANAGEMENT

PERFORMANCE MANAGEMENT

© Copyright 2015 – Keyrus 2

DIVING INTO WEBLOG DATA WITH SAS ON HADOOP

Lisa Truyers, Data Scientist Consultant at Keyrus

March 24, 2016

Logo


Project summary

WHO HAS EVER TRIED TO OPEN A 1 GB FILE ON A COMPUTER?


What is Hadoop?

Project summary

Components of the Hadoop-SAS framework

Setup to load data

Benchmarks

Lessons learned

AGENDA


PROS Open-source software framework

Storage and large-scale data processing

Easy and economic scaling

Both structured and unstructured data

Low-cost commodity hardware

Starts multiple copies of the same task for the same block of data

What is Hadoop?

51% OF COMPANIES THINKS ABOUT INTEGRATING HADOOP IN THEIR COMPANY BY 2016Philip Russom, TDWI Best Practices Report= Integrating Hadoop into Business


CONS Management and high-availability

capabilities are just starting to emerge

Data security is fragmented

MapReduce is very batch-oriented

No easy-to-use, full-feature tools for data integration, data cleansing, governance and metadata

Lacking skilled professionals

What is Hadoop?

MANAGE THE DATA AND USE ANALYTICS TO QUICKLY IDENTIFY PREVIOUSLY UNKNOWN INSIGHTS: ACCESS THE DIFFERENT TOOLS OF SAS


WHAT ARE COMPANIES DOING WITH HADOOP?The percentages mentioned here cover the whole world, not only Europe.

What is Hadoop?

What? PercentageData warehouse extensions 46 %Data exploration and discovery 46 %Data staging for data warehousing and data integration 39 %Data lake 39 %Queryable archive for non-traditional data 36 %Computational platform and sandbox for advanced analytics 33 %


WHY IS HADOOP (NOT) IMPORTANT?“Cost savings. Linear scalability. Evaluate ‘the hype’ practically. Complement BI.”

BI architect, telecom, Europe

“Reduces cost of data. New ability to query big data sets. Supply chain improvements. Predictive analytics.”Vice president, food and beverage, Asia

“Our existing infrastructure cannot handle the tenfold increase in data volumes.”Data strategy manager, hospitality, US

“It’s important to realize the potential of big data and to explore new business opportunities.”Data specialist, consulting, Asia

What is Hadoop?


What is Hadoop?

Project summary


Setup to load data

Benchmarks

Lessons learned

AGENDA


INTRODUCTIONProject summary

1. Discover web traffic data

• Discover web traffic data• Sheer volume of data makes it impossible to analyse at the moment• Prove the added value of a combined Hadoop – SAS environment

2. Lead generation

• More business oriented: scoring a neural network model takes one hour on daily basis• Reducing this time


Project summary

What is Hadoop?


Setup to load data

Benchmarks

Lessons learned

AGENDA


HADOOP COMPONENTSComponents of the Hadoop-SAS framework

HBASE PIG HIVE & HCATALOG

MAP REDUCE

HDFS

AMBARI

OOZIE

FLUME

SQOOP

NFS

WebHDFS

YARN

Base SAS & SAS/ACCESS® to Hadoop™

SAS Metadata


SAS Metadata

SAS IMSTAT for Hadoop

SAS® Visual Analytics & Statistics

SAS® LASR™ AnalyticServer

SAS® High-Performance

Analytic Procedures

SAS® Enterprise Guide®




MAP REDUCE

HDFS

AMBARI

OOZIE

FLUME

SQOOP

NFS

WebHDFS

YARN


SAS Metadata


SAS Metadata





Analytic Procedures





MAP REDUCE

HDFS

AMBARI

OOZIE

FLUME

SQOOP

NFS

WebHDFS

YARN


SAS Metadata


SAS Metadata





Analytic Procedures





MAP REDUCE

HDFS

AMBARI

OOZIE

FLUME

SQOOP

NFS

WebHDFS

YARN


SAS Metadata


SAS Metadata





Analytic Procedures





MAP REDUCE

HDFS

AMBARI

OOZIE

FLUME

SQOOP

NFS

WebHDFS

YARN


SAS Metadata


SAS Metadata





Analytic Procedures





MAP REDUCE

HDFS

AMBARI

OOZIE

FLUME

SQOOP

NFS

WebHDFS

YARN


SAS Metadata


SAS Metadata





Analytic Procedures



SAS COMPONENTSComponents of the Hadoop-SAS framework


MAP REDUCE

HDFS

AMBARI

OOZIE

FLUME

SQOOP

NFS

WebHDFS

YARN


SAS Metadata



Analytic Procedures


SAS Metadata





SAS COMPONENTSComponents of the Hadoop-SAS framework


MAP REDUCE

HDFS

AMBARI

OOZIE

FLUME

SQOOP

NFS

WebHDFS

YARN


SAS Metadata



Analytic Procedures


SAS Metadata





Project summary

What is Hadoop?


Setup to load data

Benchmarks

Lessons learned

AGENDA


FULL PROCESSSetup to load data

Day

A Partitioned, non-parsed for day-filesC Partitioned, parsed for day-files

Hour

B Partitioned, non-parsed for hour-filesD Partitioned, parsed for hour-files


Setup to load data


PROCESS CSetup to load data

Delete HIVE Table

Transfer to Hadoop Parse data Merge Loop


Project summary

What is Hadoop?


SAS-tools used in this project

Setup to load data

Benchmarks

Lessons learned

AGENDA


HADOOP COMPARED TO SERVERServer

Query test one day: 35 seconds

Parsing data on one day: 15 minutes

Parsing of one week: 4hours 30 minutes

Benchmarks

Hadoop

Query test on one day: 35 seconds

Parsing data on one day: 15 minutes

Parsing of one week: 53 minutes

MORE TIME NEEDED FOR EXTRA BENCHMARKS


Project summary

What is Hadoop?


SAS-tools used in this project

Setup to load data

Benchmarks

Lessons learned

AGENDA


Teamwork is key• Set-up Hadoop cluster with

Hadoop-experts

• Install SAS with experts from the company

SAS ON HADOOP In SAS, take your time to set the correct

variable length

Choose the strength of the cluster rationally

Create Benchmarks on both environments (server VS Hadoop) early on so a good comparison can be done and the correct decision can be taken

Data must be large enough on Hadoop to see a difference

Lessons learned

THANK YOU FOR YOUR ATTENTIONTo contact us

www.keyrus.com

[email protected]

Data & Analytics

BI congres 2016-2: Diving into weblog data with SAS on Hadoop - Lisa Truyers - Keyrus