Hortonworks Data In Motion Series Part 3 - HDF Ambari

Harnessing Data-in-Motion with Hortonworks DataFlow

HDF 2.0: Apache Ambari Integration

Ali BajwaPrincipal Partner Solutions Engineer

Yolanda DavisSenior Software Engineer

Recap

HDF 2.0: Apache NiFi, Apache MiNiFi, Apache Kafka, Apache Storm, Apache Ambari

3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Hortonworks DataFlow Manages Data-in-MotionCore

InfrastructureSources

Constrained High-latency Localized context

Hybrid – cloud / on-premises Low-latency Global context

RegionalInfrastructure

Apache NiFi, Apache MiNiFi, Apache Kafka, Apache Storm are trademarks of the Apache Software Foundation

4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

What is Apache Ambari?

Health Checks, Alerts Stacks, Views

Lifecycle controls, Rolling Restarts, Decommission/Re-

commission

Host Groups, Versioning, Compare, Revert,

Recommendations, Security Setup

Install Wizard (UI),Blueprints (API)

ConfigManagement

ExtensibilityMonitoring

ServiceManagement

ClusterProvisioning

5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

AgendaApache Ambari Integration in HDF 2.0• Deployment• Configuration• Debugging• Monitoring

Ease of Deployment

7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Deployment

Deploy Apache NiFi through:– Apache Ambari install wizard or – Operationalize via blueprints automation

Choose which nodes Apache NiFi can be installed on

So either:– Choose Apache NiFi hosts at time of

cluster install

8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Deployment– Or add Apache NiFi to existing host (post-cluster install) and then start it

9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Deployment–Can configure which user/group Apache NiFi runs as (when cluster installed or when Apache NiFi is added to existing cluster)

10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Deployment Can remove Apache NiFi service from Apache Ambari

– Note: this does not remove bits Can stop/start/configure Apache NiFi across cluster via:

– Apache Ambari UI or – Apache Ambari REST API

11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Deployment Can stop/start Apache NiFi on individual hosts via Apache Ambari UI and also via

REST API

12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Deployment

Quicklink for easy access to Apache NiFi UI – URL dynamically determined based on:

• whether SSL enabled and • which ports user configured

Demo #1: Deploy HDF 2.0

14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Demo #1: Deploy HDF 2.0

•Deploy HDF cluster using Apache Ambari Install Wizard: http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.0/bk_ambari-installation/content/ch03s01.html

–Install Apache Ambari–Install HDF mpack–Open Apache Ambari WebUI–Run through Install Wizard

Ease of Configuration

16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Configuration

Configure once via Apache Ambari (instead of repeating config on each node) Support for most Apache NiFi config files (e.g. nifi.properties, bootstrap.conf etc) Assistance for admin during Apache NiFi configuration:

– Help text displayed on hover with property descriptions– Checkboxes instead of true/false values– User friendly labels and default values– ‘Computed’ values can be automatically handled (e.g. node address)

Get benefit of Apache Ambari config features:– Config history:

• diff between versions, revert to older version etc– Update configs via Apache Ambari REST API

17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Configuration

18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Configuration•Manage host specific configs using Config groups feature

–e.g. to bump memory for Apache NiFi on one of the nodes, click Override button:

19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Configuration ...create new config group and add desired host to it

20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Configuration

‘Common’ configs grouped together and exposed in first config section (‘Advanced nifi-ambari-config’) to allow configuration of:– Ports (nonSSL, SSL, protocol)– Initial and max memory (Xms, Xmx)– Repo default dir locations (provenance, content, db, flow file)– ‘Internal’ dir location - contains files NiFi will write to

• ‘conf’ subdir for flow/tar.gz, authorizations.xml• ‘state’ subdir for internal state• Can change subdir names by prefixing the desired subdir name with

‘{nifi_internal_dir}/’– Sensitive props key (used to encrypt sensitive property values)– Zookeeper znode for nifi

21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Configuration

22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Configuration

nifi.properties exposed under ‘Advanced nifi-properties’ as key/value pairs with helptext– values replaced by Apache Ambari shown surrounded by double braces e.g.{{ }} but

can be overridden by end user– Properties can be updated or added via ‘Custom nifi-properties’ and will get written

to all nodes– ‘Computed’ values handled:

• ‘Node address’ fields replaced with each host's own FQDN

23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Configuration

Other property based config files exposed as jinja templates– Values replaced by Apache Ambari shown surrounded by double braces e.g. {{ }} but can be overridden by

end user– Properties can be added/updated and will get written to all nodes

25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Configuration Other xml based config files also exposed as jinja templates

– Values replaced by Apache Ambari shown surrounded by double braces e.g. {{ }} but can be overridden – Elements can be updated/added and will get written to all nodes

– Config files written out with either 0400 or 0600 perm

• Why? • Some property files contain plaintext passwords

Ease of Debugging

28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Debugging Logsearch integration for ease of visualizing/debugging Apache NiFi logs w/o connecting via SSH to

hosts (e.g nifi_app.log, nifi_user.log, nifi_bootstrap.log) By default, monitors FATAL,ERROR,WARN messages (for all HDF services)

29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Debugging Can view/drill into errors at component level or host level Can filter errors based on severity (fatal, error, warn, info, debug, trace) Can exclude ‘noisy’ messages to find the needle in the haystack

Ease of Monitoring

31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Monitoring Apache NiFi Service check: Used to ensure that Apache NiFi UI is up. Can also be

invoked via REST API for automation

32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Monitoring Apache NiFi Alert: Host level Apache Ambari alert when Apache NiFi process

goes down– Can temporarily be disabled by turning on maintenance mode

33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Monitoring Alerts tab allows:

– Disabling and Configuring alerts (e.g. change interval)

34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Monitoring Can setup email or

SNMP notifications to admins

35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Monitoring Apache Ambari Metrics (AMS) integration

– Service autocreates Apache Ambari reporting task, pointing to AMS collector host/port (autodetected)

36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of MonitoringThis is achieved by setting up an initial flow.xml (configurable via Amari)...

Tip: can auto-deploy any Apache NiFi flow by pasting its flow.xml here

37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Monitoring...and passing arguments via bootstrap.conf Ensures Apache NiFi autodetects updated location of AMS collector (in case it’s moved)

38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Monitoring Results in AMS dashboard

for Apache NiFi metrics, such as:– Flowfiles sent/received– MBs read/written– JVM usage/thread counts

39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Monitoring

Dashboard widgets can:– be drilled into to see

results from last 1,2,4 hours, day, week etc

– export metrics data as csv or JSON

40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Ease of Monitoring Same metrics can be viewed in Grafana dashboard

– Access via quicklink under ‘Ambari metrics’ service– Allows users to easily create custom dashboards for each component

41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Summary - Apache NiFi in Apache Ambari

• HDF available via management pack (mpack) that defines required services, stack definitions etc

• Wizard Driven Setup• Provides access to Apache NiFi configurations, start/stop/restart• Has a Reporting Task for Apache Ambari Metrics• In Cluster Mode by Default• Leverages Zookeeper on HDF for State Management (not Embedded Zookeeper)• Some Stack Advisor configurations (Ranger integration, Apache NiFi Certificate

Authority)• Audit logging via Apache Ambari Infra (Internal Solr)

* You cannot install Apache Ambari to manage an HDF cluster on a system where HDP is already installed (requires a dedicated instance)

Demo #2: Explore HDF 2.0

43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Demo #2: Explore HDF 2.0

Explore HDF 2.0– Apache NiFi configurations– Configuration groups– Metrics– Logsearch integration– Service checks– Alerts

Try it out!

45 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Try it out!

Deploy HDF cluster using either:– a) Install Wizard: http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.0/bk_ambari-installation/content/ch03s01.html

– b) Blueprints: https://community.hortonworks.com/articles/56849/automate-deployment-of-hdf-20-clusters-using-ambar.html

* Reminder: You cannot install Apache Ambari to manage an HDF cluster on a system where HDP is already installed (requires a dedicated instance)

46 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Questions? https://community.hortonworks.com/

Hortonworks Community Connection:Data Ingestion and Streaming