Apache Hadoop India Summit 2011 talk "Making Hadoop Enterprise Ready with Amazon Elastic Map/Reduce" by Simone Brunozzi

Making Hadoop Enterprise ready with Amazon Elastic Map/Reduce

Simone BrunozziTechnology Evangelist, Amazon Web Services, APACtwitter: @simonBlog: www.brunozzi.com

• What is Elastic MapReduce • Use Cases• Service Features• New Feature Announcements • Elastic MapReduce Ecosystem

AGENDA

• Enables customers to easily, securely and cost-effectively process vast amounts of data.– Spin-up 10s or 100s or even 1000s of

instances– Process 10s or 100s of Terabytes of data

• Hosted Hadoop framework running on the web-scale infrastructure of Amazon.

WHAT IS AMAZON ELASTIC MAPREDUCE

• Launch and monitor job flows• AWS Management Console• Command line interface • REST API

WHY USE AMAZON ELASTIC MAPREDUCE

• Elastic MapReduce removes MUCK from Big Data processing–Hard to manage compute clusters–Hard to tune Hadoop–Hard to monitor running Job Flows–Hard to debug Hadoop jobs–Hadoop issues prevent smooth

operation in the cloud

PROBLEMS CUSTOMERS SOLVE WITH ELASTIC MAPREDUCE

• Data mining and BI– Log processing, click stream analysis, similarities, advertizing

• Data warehousing applications• Bio-informatics (Genome analysis) • Financial simulation (Monte Carlo simulation)• File processing (resize jpegs)• Web indexing

WEB-SCALE DATA WAREHOUSING

• Hadoop 0.20• Pig 0.6• Hive 0.5• Cascading 1.1

ELASTIC MAPREDUCE – SUPPORTED CONFIGURATIONS

• Hadoop 0.18• Pig 0.3• Hive 0.4• Cascading 1.1

• Apache Hive – Batch and Interactive Mode– Support Hive Steps– Integration with Elastic MapReduce Client and Management Console– Load table partitions automatically to/from Amazon S3– Optimized data writes to Amazon S3– Reference resources such as streaming scripts located on Amazon S3– Specify an off-instance metadata store – Support variables defined directly in Hive script – Supports JDBC and ODBC connections

ELASTIC MAPREDUCE – HIVE FEATURES

• Apache Pig – Batch and interactive mode– Support Pig Steps– Integration with Elastic MapReduce Client and Management Console– Concurrent access to multiple file systems (HDFS, Amazon S3)– Reference resources in Amazon S3 directly from Pig script– Several User Defined Functions in Piggy Bank

ELASTIC MAPREDUCE – PIG FEATURES

• Enterprise customers need more flexibility– Configuring Clusters– Running Clusters– Paying for clusters

• Enterprise customers need more tools – Application development – Data analytics

• Enterprise customers need support options– Forums support is not enough

AMAZON ELASTIC MAPREDUCE FOR ENTERPRISE

• Bootstrap actions– Run arbitrary scripts before job flow begins – Run on all nodes before data processing begins – Used for

• Hadoop configuration (site-conf, Hadoop-conf, etc.)• Cluster configuration (memory, swap, etc.)• Application/packages installation (app-get install r-base)

– Several pre-defined bootstrap actions available

Amazon Elastic MapReduce features



• Enterprise customers need support options– Forum support is not enough


• Preannounce: Expand running clusters– Increase number of nodes in a running

cluster• Increase processing speed• Increasing HDFS size

Amazon Elastic MapReduce - new features

• Use Case: Increase speed of running job flows– Speed up job flow execution in response to changing requirements– Dynamically balance cost versus performance without restarting a job

PREANNOUNCE – EXPAND/SHRINK CLUSTERS

Allocate 4 instances

Expand to 25 instances


Job Flow

Time remaining:

Time remaining:14 Hours

3 Hours

Time remaining:

Job Flow

Job Flow

7 Hours

• Shrink running clusters– Decrease number of nodes in a running job flow

• Different capacity requirements from step to step• Automatically regulate capacity between steps

Amazon Elastic MapReduce - new features

• Use Case: Agile Data Warehouse Cluster– Customize cluster size to support varying resource needs (e.g., query

support during the day versus batch processing overnight)– Leverage flexibility to reduce costs and increase cluster utilization

EXPAND/SHRINK CLUSTERS



Shrink to 9 instances

Data Warehouse(Steady State)

Data Warehouse(Steady State)

Data Warehouse(Batch Processing)





AMAZON ELASTIC MAPREDUCE PRICE

WHAT IS A SPOT INSTANCE?

• Way to purchase & consume EC2 instances based on compute value

• Reduce your computing costs– Bid for unused EC2 capacity– Control your costs

• Differences from On-Demand Instances:– Request – maximum price bid– Spot Price – what you pay– Termination

M2.XLARGE INSTANCE PRICING HISTORY

Amazon EC2 On-Demand price for the same instance is $0.50

• Spot pricing support for Elastic MapReduce job flows– Specify the price you want to pay for instances– Elastic MapReduce takes care of

• Provisioning• Node addition and removal to/from the cluster

– Can mix On-Demand and Spot instances in the same cluster

Amazon Elastic MapReduce – new feature

• Use Case: Manage cost of running job flows– Start with 4 On-Demand instances of type m2.xlarge– Expand the cluster with 5 Spot Nodes

Integration with EC2 Spot



Job Flow

Time remaining:14 Hours

Time remaining:

Job Flow

7 Hours

Cost without Spot:4 instances *14 hrs * $0.50 = $28

Cost with Spot:4 instances *7 hrs * $0.50 = $13 +5 instances * 7 hrs * $0.25 = $8.75Total = $21.75

Savings: ~22%





ELASTIC MAPREDUCE ECOSYSTEM

• Ecosystem is growing– Integrated development environments for Hadoop– Tools designed for data analytics

• Broad support for Amazon Elastic MapReduce

• Big Data Intelligence software • For developers and analysts

to work faster and easier• Purpose built for all popular

Hadoop distros and versions• Tightly integrated with Elastic

MapReduce (since 2009)• Built on Karmasphere

Application Framework™– Native Hadoop client-side

platform

Karmasphere

• Rich graphical environment– Develop, debug and deploy easily

• Visualize, manipulate & diagnose– Jobs, clusters & file systems

• Broad and deep Elastic MapReduce support– Rapid development– Comprehensive profiling– Rich debugging

Karmasphere Studio

• SQL interface for ad hoc analysis• Robust Hive implementation

– Syntax checking, diagnostics, schema browser, JDBC4 compliance, multi-threaded and concurrent

• No cluster changes– Works over proxies and firewalls

• Integrated Hadoop monitoring

Professional Edition Analyst Edition

Free version fromwww.karmasphere.com

DATAMEER ANALYTICS SOLUTION

• Big data analytics leveraging native Hadoop• Extreme scale and performance• Seamless elastic scale on Amazon Elastic

MapReduce• Empowering business users• UI Driven

– no programming, no modeling, no schema, no ETL

CRM

Web Logs

Sales

Customer DataExcel Files

Social Media

DATAMEER ANALYTICS SOLUTION

Amazon Elastic MapReduce

• Corporate Overview– Founded in 1989– Largest independent public BI vendor (NASDAQ: MSTR)– Positioned in the Gartner “Leader Quadrant” for BI Platforms– Over 1 million business users at over 3,000 organizations

• The MicroStrategy 9 business intelligence platform enables mobile apps, dashboards, reporting and analytics with your business data

• Build once, deliver instantly and securely any time, to any device

MICROSTRATEGY IS A GLOBAL LEADER IN BUSINESS INTELLIGENCE

WHAT CAN YOU DO WITH MICROSTRATEGY AND AMAZON ELASTIC MAPREDUCE?

• Deliver insights to a broader range of users. – End users interact with a point-and-click interface to query data without writing

HiveQL or MapReduce jobs• Use cases:

– Mobile Apps: Floor manager accesses order details stored in Amazon Elastic MapReduce through a custom iPhone App

– Dashboards: End user starts with a Dynamic Dashboard populated from data mart or data warehouse. The user then drills to a detail report that executes in Amazon Elastic MapReduce.

– Reporting: Application developer builds a parameterized HiveQL report, then schedules it to execute. Jobs execute against Amazon Elastic MapReduce and MicroStrategy sends out exception based alerts via email to end users.

– Analysis: Application developer populates a multidimensional cache in MicroStrategy with results of a HiveQL query. End user uses MicroStrategy’s web interface to slice-and-dice through results without going back to Hadoop.

HOW CAN I LEARN MORE?

• Try it!– Free MicroStrategy software is available at:

http://www.microstrategy.com/freereportingsoftware• Get More information about Microstrategy solutions

for Amazon Elastic MapReduce http://aws.amazon.com/solutions/solution-providers/microstrategy

http://www.microstrategy.com/freereportingsoftware

http://aws.amazon.com/solutions/solution-providers/microstrategy

http://aws.amazon.com/solutions/solution-providers/microstrategy



• Enterprise customers need more support options– Forums support is not enough


ELASTIC MAPREDUCE - SUPPORT



• Enterprise customers need more support options– Forums support is not enough


Making Hadoop Enterprise ready with Amazon Elastic Map/Reduce

Simone BrunozziTechnology Evangelist, Amazon Web Services, APACtwitter: @simonBlog: www.brunozzi.com

Documents

Apache Hadoop India Summit 2011 talk "Making Hadoop Enterprise Ready with Amazon Elastic Map/Reduce" by Simone Brunozzi