Upload
alon-torres
View
95
Download
0
Embed Size (px)
Citation preview
Standalone Spark DeploymentFor Stability and Performance
Totango❖ Leading Customer Success Platform
❖ Helps companies retain and grow their customer base
❖ Advanced actionable analytics for subscription and recurring revenue
❖ Founded @ 2010
❖ Infrastructure on AWS cloud
❖ Spark for batch processing
❖ ElasticSearch for serving layer
About Me
Romi Kuntsman
Senior Big Data Engineer @ Totango
Working with Apache Spark since v1.0
Working with AWS Cloud since 2008
Spark on AWS - first attempts❖ We tried Amazon EMR (Elastic MapReduce) to install Spark on YARN
➢Performance hit per application (starts Spark instance for each)
➢Performance hit per server (running services we don't use, like HDFS)
➢Slow and unstable cluster resizing (often stuck and need to recreate)
❖We tried spark-ec2 script to install Spark Standalone on AWS EC2 machines
➢Serial (not parallel) initialization of multiple servers - slow!
➢Unmaintained scripts since availability of Spark on EMR (see above)
➢Doesn't integrate with our existing systems
Spark on AWS - road to success❖ We decided to write our own scripts to integrate and control everything
❖Understood all Spark components and configuration settings
❖Deployment based on Chef, like we do in all servers
❖Integrated monitoring and logging, like we have in all our systems
❖Full server utilization - running exactly what we need and nothing more
❖Cluster hanging or crashing no longer happens
❖Seamless cluster resize without hurting any existing jobs
❖Able to upgrade to any version of Spark (not dependant on third party)
What we'll discuss❖Separation of Spark Components
❖Centralized Managed Logging
❖Monitoring Cluster Utilization
❖Auto Scaling Groups
❖Termination Protection
❖Upstart Mechanism
❖NewRelic Integration
❖Chef-based Instantiation
Data w/ RomiOps w/ Alon
Separation of Components❖Spark Master Server (single)
➢Master Process - accepts requests to start applications
➢History Process - serves history data of completed applications
❖Spark Slave Server (multiple)
➢Worker Process - handles workload of applications on server
➢External Shuffle Service - handles data exchange between workers
➢Executor Process (one per core - for running apps) - runs actual code
Configuration - Deploy Spread Out❖spark.deploy.spreadOut (SPARK_MASTER_OPTS)
➢true = use cores spread across all workers
➢false = fill up all worker cores before getting more
Configuration - Cleanup❖spark.worker.cleanup.* (SPARK_WORKER_OPTS)
➢.enabled = true (turn on mechanism to clean up app folders)
➢.interval = 1800 (run every 1800 seconds, or 30 minutes)
➢.appDataTtl = 1800 (remove finished applications after 30 minutes)
❖We have 100s of applications per day, each with it's jars and logs
❖Rapid cleanup is essential to avoid filling up disk space
❖We collect the logs before cleanup - details in following slides ;-)
❖Only cleans up files of completed applications
External Shuffle Service❖Preserves shuffle files written by executors
❖Servers shuffle files to other executors who want to fetch them
❖If (when) one executor crashes (OOM etc), others may still access it's shuffle
❖We run the shuffle service itself in a separate process from the executor
❖To enable: spark.shuffle.service.enable=true
❖Config: spark.shuffle.io.* (see documentation)
Logging - components❖ Master Log (/logs/spark-runner-org.apache.spark.deploy.master.Master-*)
➢Application registration, worker coordination
❖History Log (/logs/spark-runner-org.apache.spark.deploy.history.HistoryServer-*)
➢Access to history, errors reading history (e.g. I/O from S3, not found)
❖Worker Log (/logs/spark-runner-org.apache.spark.deploy.worker.Worker-*)
➢Executor management (launch, kill, ACLs)
❖Shuffle Log (/logs/org.apache.spark.deploy.ExternalShuffleService-*)
➢External Executor Registrations
Logging - applications❖Application Logs (/mnt/spark-work/app-12345/execid/stderr)
➢All output from executor process, including your own code
❖Using LogStash to gather logs from all applications togetherinput {
file { path => "/mnt/spark-work/app-*/*/std*" start_position => beginning
}}filter {
grok { match => [ "path", "/mnt/spark-work/%{NOTSPACE:application}/.+/%{NOTSPACE:logtype}" ]
}}output {
file { path => "/logs/applications.log" message_format => "%{application} %{logtype} %{message}"
}}
Monitoring Cluster Utilization❖ Spark Reports Metrics (Codahale) through Graphite
➢Master metrics - running application and their status
➢Worker metrics - used cores, free cores
➢JVM metrics - memory allocation, GC
❖We use Anodot to view and track
metrics trends and anomalies
And now, to the Ops side...
Alon Torres
DevOps Engineer @ Totango
Auto Scaling Group Components❖Auto Scaling Group
➢Scale your group up or down flexibly
➢Supports health checks and load balancing
❖Launch Configuration
➢Template used by the ASG to launch instances
➢User Data script for post-launch configuration
❖User Data
➢Install prerequisites and fetch instance info
➢Install and start Chef client
➢Sanity checks throughout
Launch Configuration
Auto Scaling Group
EC2InstanceEC2
InstanceEC2InstanceEC2
InstanceEC2InstanceEC2
Instance
User Data
Auto Scaling Group resizing in AWS❖ Scheduled
➢Set the desired size according to a specified schedule
➢Good for scenarios with predictable, cyclic workloads.
❖Alert-Based
➢Set specific alerts that trigger a cluster action
➢Alerts can monitor instance health properties (resource usage)
❖Remote-triggered
➢Using the AWS API/CLI, resize the cluster however you want
Resizing the ASG with Jenkins❖We use schedule-based Jenkins jobs that utilize the AWS CLI
➢Each job sets the desired Spark cluster size
➢Makes it easy for our Data team to make changes to the schedule
➢Desired size can be manually overridden if needed
Termination Protection❖When scaling down, ASG treats all nodes as equal termination candidates
❖We want to avoid killing instances with currently running jobs
❖To achieve this, we used a built-in feature of ASG - termination protection
❖Any instance in the ASG can be set as protected, thus preventing
termination when scaling down the cluster.if [ $(ps -ef | grep executor | grep spark | wc -l) -ne 0 ]; then
aws autoscaling set-instance-protection --protected-from-scale-in …
fi
Upstart Jobs for Spark❖ Every spark component has an upstart job the does the following
➢Set Spark Niceness (Process priority in CPU resource distribution)
➢Start the required Spark component and ensure it stays running
■The default spark daemon script runs in the background
■For Upstart, we modified the script to run in the foreground
❖ nohup nice -n "$SPARK_NICENESS"…&
vs
❖ nice -n "$SPARK_NICENESS" ...
NewRelic Monitoring❖ Cloud-based Application and Server monitoring
❖Supports multiple alert policies for different needs
➢Who to alert, and what triggers the alerts
❖Newly created instances are auto - assigned the default alert policy
Policy Assignment using AWS Lambda❖Spark instances have their own policy in NewRelic
❖Each instance has to ask NewRelic to be reassigned to the new policy
➢Parallel reassignment requests may collide and override each other
❖Solution - during provisioning and shutdown, we do the following:
➢Put a record in an AWS Kinesis stream that contains their hostname and
their desired NewRelic policy ID
➢The record triggers an AWS Lambda script that uses the NewRelic API to
reassign the hostname given to the policy ID given
Chef❖Configuration Management Tool, can provision and configure instances
➢Describe an instance state as code, let chef handle the rest
➢Typically works in server/client mode - client updates every 30m
➢Besides provisioning, also prevents configuration drifts
❖Vast amount of plugins and cookbooks - the sky's the limit!
❖Configures all the instances in our DC
Spark Instance Provisioning❖ Setup Spark
➢Setup prerequisites - users, directories, symlinks and jars
➢ Download and extract spark package from S3
❖Configure termination protection cron script
❖Configure upstart conf files
❖Place spark config files
❖Assign NewRelic policy
❖Add shutdown scripts
➢Delete instance from chef database
➢Remove from NewRelic monitoring policy
Questions?❖ Alon Torres, DevOps
https://il.linkedin.com/in/alontorres
❖Romi Kuntsman, Senior Big Data Engineer
https://il.linkedin.com/in/romik
❖Stay in touch!
Totango Engineering Technical Blog
http://labs.totango.com/