View
8.277
Download
2
Category
Preview:
Citation preview
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rahul Pathak, AWS
Scott Donaldson, FINRA
Clayton Kovar, FINRA
October 2015
Amazon EMR Deep Dive
& Best Practices
BDT305
What to expect from the session
• Update on the latest Amazon EMR release
• Information on advanced capabilities of Amazon EMR
• Tips for lowering your Amazon EMR costs
• Deep dive into how FINRA uses Amazon EMR and
Amazon S3 as their multi-petabyte data warehouse
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rahul Pathak, Sr. Mgr. Amazon EMR
(@rahulpathak)
October 2015
Amazon EMRDeep Dive & Best Practices
Amazon EMR
• Managed clusters for Hadoop, Spark, Presto, or any other applications in the Apache/Hadoop stack
• Integrated with the AWS platform via EMRFS – connectors for Amazon S3, Amazon DynamoDB, Amazon Kinesis, Amazon Redshift, and AWS KMS
• Secure with support for AWS IAM roles, KMS, S3 client-side encryption, Hadoop transparent encryption, Amazon VPC, and HIPAA-eligible
• Built in support for resizing clusters and integrated with the Amazon EC2 spot market to help lower costs
New Features
EMR Release 4.1
• Hadoop KMS with transparent HDFS encryption support
• Spark 1.5, Zeppelin 0.6
• Presto 0.119, Airpal
• Hive, Oozie, Hue 3.7.1
• Simple APIs for launch and configuration
Intelligent Resize
• Incrementally scale up based on available capacity
• Wait for work to complete before resizing down
• Can scale core nodes and HDFS as well as task nodes
Amazon S3 as your persistent data store
• Separate compute and storage
• Resize and shut down Amazon EMR clusters with no data loss
• Point multiple Amazon EMR clusters at the same data in Amazon S3
• Easily evolve your analytic infrastructure as technology evolves
EMR
EMR
Amazon
S3
EMRFS makes it easier to use Amazon S3
• Read-after-write consistency
• Very fast list operations
• Error handling options
• Support for Amazon S3 encryption
• Transparent to applications: s3://Amazon
S3
Going from HDFS to Amazon S3
CREATE EXTERNAL TABLE serde_regex(
host STRING,
referer STRING,
agent STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
)
LOCATION ‘samples/pig-apache/input/'
Going from HDFS to Amazon S3
CREATE EXTERNAL TABLE serde_regex(
host STRING,
referer STRING,
agent STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
)
LOCATION 's3://elasticmapreduce.samples/pig-apache/input/'
Amazon S3EMRFS metadata
in Amazon DynamoDB
List and read-after-write consistency
Faster list operations
Consistent view and fast listing using
the optional EMRFS metadata layer
*Tested using a single node cluster with a m3.xlarge instance.
Number of
objects
Without
consistent view
With
consistent view
1,000,000 147.72 29.70
100,000 12.70 3.69
EMRFS client-side encryption
Amazon S3
Am
azon S
3 e
ncry
ption c
lients
EM
RF
S e
nable
d fo
r
Am
azon S
3 c
lient-s
ide e
ncry
ptio
n
Key vendor (AWS KMS or your custom key vendor)
(client-side encrypted objects)
HDFS is still there if you need it
• Iterative workloads
• If you’re processing the same dataset more than once
• Consider using Spark & RDDs for this too
• Disk I/O intensive workloads
• Persist data on Amazon S3 and use S3DistCp to copy to/from
HDFS for processing
File formats
Row oriented• Text files
• Sequence files
• Writable object
• Avro data files
• Described by schema
Columnar format• Object Record Columnar (ORC)
• Parquet
Logical table
Row oriented
Column oriented
Factors to consider
Processing and query tools• Hive, Impala and Presto
Evolution of schema• Avro for schema and Presto for storage
File format “splittability”• Avoid JSON/XML Files. Use them as records
Encryption requirements
File sizes
Avoid small files
• Anything smaller than 100MB
Each mapper is a single JVM
• CPU time is required to spawn JVMs/mappers
Fewer files, matching closely to block size
• fewer calls to S3
• fewer network/HDFS requests
Dealing with small files
Reduce HDFS block size, e.g. 1MB (default is 128MB)
• --bootstrap-action s3://elasticmapreduce/bootstrap-
actions/configure-hadoop --args “-m,dfs.block.size=1048576”
Better: Use S3DistCp to combine smaller files together
• S3DistCp takes a pattern and target path to combine smaller
input files to larger ones
• Supply a target size and compression codec
Compression
Always compress data files On Amazon S3
• Reduces network traffic between Amazon S3 and Amazon EMR
• Speeds Up Your Job
Compress mappers and reducer output
Amazon EMR compresses inter-node traffic with LZO with
Hadoop 1, and Snappy with Hadoop 2
Choosing the right compression
• Time sensitive, faster compressions are a better choice
• Large amount of data, use space efficient compressions
• Combined Workload, use gzip
Algorithm Splittable? Compression ratioCompress +
decompress speed
Gzip (DEFLATE) No High Medium
bzip2 Yes Very high Slow
LZO Yes Low Fast
Snappy No Low Very fast
Cost saving tips for Amazon EMR
Use S3 as your persistent data store – query it using
Presto, Hive, Spark, etc.
Only pay for compute when you need it
Use Amazon EC2 Spot instances to save >80%
Use Amazon EC2 Reserved instances for steady
workloads
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scott Donaldson, Senior Director
Clayton Kovar, Principal Architect
EMR &
Interactive
Analytics
EMR is Ubiquitous in our architecture
Data Marts
(Amazon
Redshift)
Query Cluster
(EMR)
Query Cluster
(EMR)
Auto Scaled
EC2
Analytics
App
Normalization
ETL Clusters
(EMR)
Batch Analytic
Clusters
(EMR)
Adhoc Query
Cluster (EMR)
Auto Scaled
EC2
Analytics
App
Users Data
ProvidersAuto Scaled
EC2
Data
Ingestion
Services
Optimization
ETL Clusters
(EMR)
Shared Metastore
(RDS)
Query Optimized
(S3)
Auto Scaled EC2
Data
Catalog
& Lineage
Services
Reference Data
(RDS)
Shared Data Services
Auto Scaled
EC2
Cluster Mgt
& Workflow
Services
Source of
Truth (S3)
It starts with the data
S3 is your durable system of recordSeparate your compute and storage
Shutdown your cluster when not in use
Share data among multiple clusters
Fault tolerance and disaster recovery
Use EMRFS for consistent view
Partition your data for performanceOptimize for your query use cases and access patterns
Larger files >256MB are more efficient
Compact small files into >100MB
FINRA Data Manager orchestrates
data between storage and compute
clustersUnified catalog
Manage EMR clusters
Track usage & lineage
Job orchestration
File formats & compression
Text format for archival copies on S3 & Amazon Glacier
Select compression algorithm for best fitWe wanted high compression for archive copy
Select a row or columnar format for performanceSequence or AVRO
ORC, Parquet, RC File
Columnar Benefits:Predicate pushdown
Skip unwanted columns
Serve multiple query engines: Hive, Presto, Spark
Avoid bloated formats with repetitive markup (e.g. XML)
Our partition and query strategy
Data
received
as:
Users
query
by:
Symbol Group 1
Symbol Group 2
Symbol Group 3
…
Symbol Group 100
Symbol & Firm Query
Late
Data
All late
records
scanned
for all
queries
On Time Data(Processing Date = Event Date)
99.97% of all records are on time
Symbol Only QueryF
irm O
nly
Qu
ery
Example hive table creation
create external table if not exists NEW_ORDERS (…)partitioned by (EVENT_DT DATE, HASH_PRTN_NB SMALLINT)stored as orclocation 's3://reinvent/new_orders/'tblproperties ("orc.compress"="SNAPPY");
alter table NEW_ORDERS add if not existspartition (event_dt='2015-10-08', hash_prtn_nb=0) location
's3://reinvent/new_orders/event_dt=2015-10-08/hash_prtn_nb=0/’
…
partition (event_dt='2015-10-08', hash_prtn_nb=1000) location 's3://reinvent/new_orders/event_dt=2015-10-08/hash_prtn_nb=1000/’
;
Each record’s hash partition number is calculated by
((pmod(hash(symbol), 100) * 10) + pmod(firm, 10))
Partitions are great, but beware…
select … from NEW_ORDERS where
EVENT_DT between '2015-10-06' and '2015-10-09’
and FIRM = 12345
and (pmod(HASH_PRTN_NB, 10) = pmod(12345, 10)
or HASH_PRTN_NB = 1000) -- 1000 is always read
Using PMOD around the hash_prtn_nb prevents Hive from using a targeted
query on the metastore resulting in millions of partitions returned for pruning
Optimized query with enumeration
select … from NEW_ORDERS where
EVENT_DT >= '2015-10-06' and EVENT_DT <= '2015-10-09’
and FIRM = 12345
and (HASH_PRTN_NB = 5 or HASH_PRTN_NB = 15
… or HASH_PRTN_NB = 985 or HASH_PRTN_NB = 995
or HASH_PRTN_NB = 1000) -- 1000 is always read
Using an IN clause was insufficient to avoid the pruning issue
Explicitly enumerating all partitions vastly improved query planning time
Data security
Required to have encryption of all data both at-rest, and in-transit
S3 server-side encryption was evaluated and determined to be suitable for purpose
Encrypt ephemeral storage on Master, Core, and Task nodes Use a custom bootstrap action with LUKS with a random, memory only key
Task nodes don’t have HDFS but Mapper and Reducer temporary files need to also be encrypted
Lose the server, lose the data – Remember S3 is our source of truth
Use security groups to ensure only the client applications connect to the Master node
Hive authentication/authorization was not necessary for our usage scenarios
Evaluating transparent encryption (Hadoop 2.6+) in HDFS
Selection of the fittest
HDFS was cost prohibitive for our use casesNeed 30 D2.8XL’s just to store two of our tables: ~$1.5M/yr on HDFS vs ~$120K/yr on S3
Need 90 D2.8XL’s to store all queryable data: ~$4.5M/yr on HDFS vs. $360K/yr on S3
Data locality is desirable but not practical for our scale EMR & S3 with partitioned data is a great fit
Tuned queries & data structures on S3 take ~2X if on HDFS under perfect locality conditions
Localize data into HDFS on Core nodes using S3DistCp if making 3 or more passes
Consider tiered storageExternal tables in Hive can have a blend of some partitions in HDFS and others in S3
Introduces operational complexity for partition maintenance
Doesn’t play well with shared metastore for multiple clusters
Darwin rules: Adaptation
Take advantage of new instance types
Find the right instance type(s) for your workload
Prefer a smaller cluster of larger nodes: e.g. 4XL
With millions of partitions, more memory is needed for the Master node (HS2)
Use CLI based scripts rather than console → Infrastructure is code
Node Type Before After
Master 1 - R3.4XL 1 - R3.2XL
Core 40 - M3.2XL 10 - C3.4XL
Task (peak) 100 - M3.2XL 35 - C3.4XL
Right size your cluster
Transient use cases: ETL and batch analyticsSize cluster to complete within ten minutes of an hour boundary to optimize $$
Use Spot when you have flexible SLA to save $$
Use On Demand or Reserved to meet SLA at predictable cost
Always On use case: Interactive analyticsSize Core based on HDFS needs (statistics, logging, etc)
Reserve Master and Core nodes
Resize # of Task nodes as demand changes
Use Spot on Task nodes to save $$
Keep a ratio of Core to Task of 1:5 to avoid bottlenecks
Consider bidding Spot above the On Demand price to ensure greater stability
One metastore to rule them all
Consider creating a shared hive metastore service
Fault tolerance & DR with Multi-AZ RDS
Offload metastore hydration of tables and partitionsTransient clusters initialize faster
Millions of partitions/day can take >7 min/day per table
Avoid duplicative effort by separate development teams
Separate metastores are needed for Hive 0.13.1, Hive 1.0 and PrestoHowever, you can locate them all on a single RDS instance
Utilize FINRA Data Management services to orchestrate metastore updatesRegister new tables and partitions as the data arrives via notifications
Monitor, learn, and optimize
Utilize workload management: Fair Scheduler
Refactor your code as necessary to remove bottlenecks
Optimize transient clusters, size to execute workload 10 minutes from an hour boundary
Set hive.mapred.reduce.tasks.speculative.execution = FALSE when writing to external tables
in S3 via Map Reduce
Use broadcast joins when joining small tables (SET hive.auto.convert.join=true).
EMR Step API works fore simple job queuing; use Oozie for more complex jobs
The impact
Removed obstacles
“Before data analysis of this magnitude required intervention from the technology team.”
Lowered the cost of curiosity
“Analysts are able to quickly obtain a full picture of what happens to an order over time,
helping to inform decision making as to whether a rule violation has occurred.”
Elasticity allows us to process years of data in days as opposed to months and save
money by using Spot market
Separately optimize batch and interactive workloads without compromise
Increased teams delivery velocity
Recap
Use Amazon S3 as your durable system of record
Use transient clusters as much as possible
Resize clusters and use the Spot to more efficiently manage capacity, performance, & cost
Move to new instance families to take advantage of performance
Monitor to determine when to resize or change instance types
Share a persistent Hive metastore in RDS among multiple EMR clusters
Be prepared to switch your query engine or execution framework in the future
Budget time to experiment for new tools & engines at scale that weren’t possible before
Related sessions
BDT208 - A Technical Introduction to Amazon Elastic MapReduce Thursday, Oct 8, 12:15 PM - 1:15 PM– Titian 2201B
BDT303 - Running Spark & Presto on the Netflix Big Data Platform
Thursday, Oct 8, 11:00 AM - 12:00 PM– Palazzo F
BDT309 - Best Practices for Apache Spark on Amazon EMR
Thursday, Oct 8, 5:30 PM - 6:30 PM– Palazzo F
BDT314 - Big Data/Analytics on Amazon EMR & Amazon Redshift
Thursday, Oct 8, 1:30 PM - 2:30 PM– Palazzo F
Recommended