Managing Apache HAWQwith Apache AMBARI

Apache Ambari Meetup - June 27, 2016

Alexander DenissovBhuvnesh Chaudhary

Mithun Mathew

Apache HAWQ(incubating) Apache Ambari

Hadoop-native SQL query engine and advanced analytics MPP database that offers:

1

2

3

4

5

interactive query execution

high performance

machine learning algorithms

tools for Data Analysts and Data Scientists

processing for large and complex data sets

APACHE HAWQ (incubating)

APACHE HAWQ (incubating) ARCHITECTURE

HAWQ - AMBARI INTEGRATION SCOPE

Installation and configuration

Topology and configuration recommendations and validations

Kerberos and High Availability support

HAWQ Master - HAWQ Standby failoverService and Component Alerts

Visual Widgets

HAWQ - AMBARI INTEGRATION EFFORT

PraisesAmbari’s pluggable architecture makes integrations like this possible and easyKerberos setup is fully metadata driven — major kudos!

ChallengesHAWQ is not part of the HDP stack and is not available in Ambari out-of-the boxAdvanced features and wizards require JavaScript code modifications

Driven by the team of engineers at PivotalDeveloped integrations from basic to more advanced

Invaluable support from Ambari CommunityTHANK YOU!

RECOMMEND SERVICE TOPOLOGY

VALIDATE SERVICE TOPOLOGY

RECOMMEND AND VALIDATE CONFIGS

HAWQ SERVICE SUMMARY PAGE

HAWQ SERVICE ACTIONS

ACTIVATE HAWQ STANDBY WIZARDActivate HAWQ Standby Wizard (Manual Operation)

ACTIVATE HAWQ STANDBY WIZARD

HAWQ Standby Masterpromoted to HAWQ Master

Add HAWQ Standby Masteraction becomes visible

HAWQ ALERTSStatus of HAWQ Components

Communication issuesbetween HAWQ Components

HAWQ AMBARI FUTURE INTEGRATION

Support automated upgrade independent of stackOngoing related work: AMBARI-14854, AMBARI-12885

Ambari requires service restart for pushing configuration changes. What if, the service can reload configurations without restart? Ongoing related work: AMBARI-17241

HAWQ Upgrade

Dynamic Configuration Reload

Display query historyManage resource queues

HAWQ View

Currently Ambari does not support configuration changes without restarting service

Some parameters do NOT require restart!

HDFS dfs.heartbeat.interval, dfs.namenode.heartbeat.recheck-interval

HAWQ default_hash_table_bucket_number, hawq_rm_memory_limit_perseg

DYNAMIC CONFIGURATION RELOAD

Currently Ambari does not support configuration changes without restarting service

Some parameters do NOT require restart!

HDFS dfs.heartbeat.interval, dfs.namenode.heartbeat.recheck-interval

HAWQ default_hash_table_bucket_number, hawq_rm_memory_limit_perseg

DOWNTIME!!!Consequence of Restarting the Service:

DYNAMIC CONFIGURATION RELOAD

No more DOWNTIME!!!

DYNAMIC CONFIGURATION RELOAD

resources/common-services/HAWQ/2.0.0/configurations/hawq-site.xml

<property>

<name>default_hash_table_bucket_number</name>

<value>6</value>

<supports-reload>true</supports-reload>

</property>

resources/common-services/HAWQ/2.0.0/package/scripts/hawqmaster.py

classHawqMaster(Script):

defstart(self,env):

…

defstop(self,env):

…

defreload(self,env):

self.configure(env)

Execute(‘hawqmasterreload’,…)

HOW TO USE

Ambari UI

Show ReloadButton

Desired Configs

Updater

HeartBeatProcessor

ServiceComponentHost(updates requires_reload)

Request Handling and Execution

Ambari Agent

Reload Method

Ambari Server

Ambari Web

Ambari Agent

Reload Method

Ambari Agent

Reload Method

Ambari Agent

Reload Method

REST API

POST

PUT

GET

Heartbeats

invalidatesrequires_reload

user specifies

COLLABORATION DIAGRAM

Feedback?

AMBARI-17241

Reload vs Restart - Are they mutually exclusive?

THINGS TO DECIDE

Seriously, purple?