Hadoop a Highly Available and Secure Enterprise Data Warehousing solution

www.edureka.co/r-for-analytics

www.edureka.co/hadoop-admin

Hadoop : A Highly Available and Secure Enterprise Data warehousing Solution

Slide 2Slide 2 www.edureka.co/hadoop-admin

At the end of this webinar we will Know about:

What is Big Data

Why do Enterprise care about Big Data

Why your DWH needs Hadoop?

Security in Hadoop

How Hadoop maintains high Availability

Data warehousing tools in Hadoop

Agenda

What is Big Data

What is Wrong with our traditional DWH Solutions

Storing Unstructured data like images and video

Processing images and video

Storing and processing other large files

PDFs, Excel files

Processing large blocks of natural language text

Blog posts, job ads, product descriptions

Processing semi-structured data

CSV, JSON, XML, log files

Sensor data

When RDBMS Makes no Sense?

Ad-hoc, exploratory analytics

Integrating data from external sources

Data cleanup tasks

Very advanced analytics (machine learning)

When RDBMS Makes no Sense?

It is:

– Unstructured

– Unprocessed

– Un-aggregated

– Un-filtered

– Repetitive

– Low quality

– And generally messy.

Oh, and there is a lot of it.

Big Problems with Big Data

Storage capacity

Storage throughput

Pipeline throughput

Processing power

Parallel processing

System Integration

Data Analysis

Scalable storage

Massive Parallel Processing

Ready to use tools

Technical Challenges

Too many channels for data

Technical Challenges

Why do Enterprise care about Big Data

You said RDBMS does not have solution

for Big Data, Then who has???

I Have The solution for Big Data Problem

Hadoop

Hadoop : The Savior

How Hadoop differs from RDBMS

Hadoop can store all types of data in it so that you have flexibility of analyzing all types of data.

You can drill down the big data to find even the rare insight which was not possible earlier.

First Load the data then do whatever you want to do.

This is Possible because of the cheap storage and distributed HDFS.

Hadoop Is The New DWH Solution

• This is ETL• Before loading you should

transform data in particular format

• This puts an restriction on the type of data that can be stored

First Load the data then do whatever you want to do.

This is Possible because of the cheap storage and distributed HDFS.

Hadoop Is The New DWH Solution

• This is ETL• Before loading you should

transform data in particular format

• This puts an restriction on the type of data that can be stored

• This is ELT• There is no need to transform

the data beforehand• You can have all kind of data on

board• Freedom to work with all data

Hadoop is the new Data Warehouse for all kind of BI requirements.

Hadoop Does ELT Not ETL

Core Features of Hadoop

Hadoop Is Fault Tolerant And Super Consistent

Maintaining High Availability(HA)

In Distributed Computing, failure is a norm, which means YARN should have acceptable amount of availability

NameNode - No Horizontal Scale

NameNode - No High Availability

DataNode

Client get Block Locations

Read Data

NameNodeNS

Block Management

Secondary NameNode:

"Not a hot standby" for the NameNode

Connects to NameNode every hour*

Housekeeping, backup of NemeNode metadata

Saved metadata can build a failed NameNode

SecondaryNameNode

NameNode

metadata

Single PointFailure

You give me metadata

every hour, I will make it

secure

NameNode – Single Point of Failure

Node Manager

Resource Manager

Shared edit logs

All name space edits logged to shared NFS storage; single writer

(fencing)

Read edit logs and applies to its own namespace

Secondary Name Node

DataNode

Standby NameNode

Active NameNode

ContainerApp

Master

Node Manager

DataNode

ContainerApp

Master

Data Node

Client

DataNode

ContainerApp

Master

Node Manager

DataNode

ContainerApp

Master

Node Manager

NameNode High Availability

Next Generation MapReduce

HDFS HIGH AVAILABILITY

http://hadoop.apache.org/docs/stable2/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html

Hadoop 2.0 Cluster Architecture - HA

Achieving HDFS and YARN High Availability

Hadoop is Secure

Security

Service-level authorization and web proxy capabilities in YARN.

Access Control Lists(ACL) : The Hadoop Distributed File System (HDFS) implements a permissions model for files and directories that shares much of the POSIX model

Security – Simple Flow

Security Risks

Insufficient Authentication Do not authenticate users services

No Privacy and No Integrity Insecure Network Transport No Message level security

Arbitrary Code Execution No User verification for MapReduce code

execution, malicious users could submit a job

Client Job Tracker

Task Tracker

Managing users, permissions , quotas, etc …

Checking Resources Usage And Users Permissions

Demo on ACL

Hadoop provides traditional SQL interface as well asNoSQL Interface foe data storage

Hive ??

Hive Architecture

Hbase and its Architecture??

Hive and HBase Integration

Questions

Your feedback is vital for us, be it a compliment, a suggestion or a complaint. It helps us to make your experience better!

Please spare few minutes to take the survey after the webinar.

Survey

Hadoop a Highly Available and Secure Enterprise Data Warehousing solution

Technology

india logistics & warehousing india logistics & warehousing

How to Deploy a Secure, Highly-Available Hadoop Platform · How to Deploy a Secure, Highly-Available Hadoop Platform . Agenda ! Environment ! Automation ... puppetdb (Database of

Infrastructure - Amazon Web Services APPS ANALYTICS APP SERVICES MOBILE SERVICES DEVELOPMENT & OPERATIONS Data Warehousing Hadoop / Spark …

Data warehousing con hadoop y el paradigma map reduce

Centralized HADOOP Secure Data Store with Data Integration ... · traditional data warehousing and business intelligence tools the customer wanted to implement a cost-effective and

HIVE Data Warehousing Analytics on Hadoop Facebook Data Team

Hadoop Performance Tuning A case studystatic.stevereads.com/papers_to_read/hadoop... · Hadoop Highly conﬁgurable commodity cluster compung programming framework Hides messy details

Hadoop-GIS: A High Performance Spatial Data Warehousing

LINKED (OPEN) DATA - unige.it...(data-warehousing) Virtually materialized approaches Federated Systems (Exported schema) MapReduce/ Hadoop Shared-memory architectures (Message-Passing,

HIVE: Data Warehousing & Analytics on Hadoop

PEPPERDATA ON A HIGHLY-TUNED HADOOP CLUSTERsvisit.pepperdata.com/rs/921-KBZ-105/images... · Hadoop Tuning Best Practices Anyone who has been working with Hadoop for any period of

3.droppdf.com3.droppdf.com/files/IpRUa/practical-hadoop-security.pdf · Lakhe Shelve in Databases/Data Warehousing User level: Intermediate–Advanced SOURCE CODE ONLINE BOOKS FOR

DATA ANALYTICS & MACHINE LEARNING · - Predict the characteristics of high LTV customers and helps in customer segmentation. ... - Python - SAS 2. Data Warehousing - Hadoop - SQL

THE LEADING PUBLICATION FOR BI, DATA WAREHOUSING, AND …/media/6397D94CDAA04B4BB464390F54ED... · 2014. 9. 15. · the Hadoop framework components most relevant to the data warehouse

Data warehousing on Hadoop

Self-Service Analytics Data Warehousing on Hadoop Data ... · PDF fileBig Data Consulting (Spark, Hadoop) Data Hub / Data Warehouse ... Issues Legacy Architecture p: +353 1 254 2897

Big Data Analytics for Fraud Detection · Data warehousing & Big Data: ETL, Data quality management, Storage, Metadata management, OLAP, Hadoop Business intelligence, reports & dashboards,

DATA WAREHOUSING · Data Vault, Hadoop, SuperNova, Data Science, Big Data Analytics, ... “Data Virtualization for Business Intelligence Systems”, alsook tientallen whitepapers

3 Hadoop? Cloud data warehousing? Machine learning? NoSQL?

Apache Hive - cs.cmu.edu › ~pavlo › courses › fall2013 › static › slides › hive.pdf · Apache HIVE Data Warehousing & Analytics on Hadoop Hefu Chai. What is HIVE? •A