Confidential

BEST PRACTICES FOR MANAGING MULTI-TENANT HADOOP

Confidential2

TRUSTEDby over 10,000

companies as their big data app platform

BACKEDby top Silicon Valley

investors True Ventures, Rembrandt

VP, Bain Capital

FOUNDED in 2008, with

headquarters in San Francisco

Confidential

WHY NOW?

3

As Big Data applications become the engine driving innovation and new revenue models, they graduate to 1st class citizens demanding higher standards of quality, reliability, and manageability

Confidential

PERFORMANCE MANAGEMENT FOR BIG DATA APPLICATIONS

your big data apps

MONITOR to resolve

issues fasterbig data apps

more effectively

MANAGE

COLLABORATE

Confidential5

ConfidentialConfidential6

FUNCTIONAL WALK-THROUGH

Confidential 7Confidential & Proprietary

ACHIEVING OPERATIONAL EXCELLENCE

• Performance monitoring and visibility for all big data applications- See everything in one place with a single view of all data applications real-time performance

metrics across the entire enterprise• Instantly understand the business context of a problem

- Knowing you have a problem is one thing but understanding the end to end & downstream dependencies, the quality and service guarantees, who is impacted, the owner(s) and the business criticality of the failure is game changing

• Easily collaborate across teams to resolve issues faster- Multi-tenant architectures are inherently chaotic. Accelerate issue resolution by delivering the right

information to the appropriate individuals and teams so everyone is informed and can work together to resolution

• Manage the business of big data- Get the operational visibility and controls you need to effectively manage priorities, policies,

capacity, compliance and utilization.

Balancing a fully operational data processing operation and customer demands is challenging

1

2

3

4

Confidential 8Confidential & Proprietary

PERFORMANCE MONITORING & VISIBILITY

Comprehensive view of all your data processing execution Fully visualize your entire data pipeline

See all successful, failed, pending processes

…quickly segment your aggregate view by app name, team , cluster etc…

Confidential 9Confidential & Proprietary

PERFORMANCE MONITORING & VISIBILITY

Comprehensive view of all your data processing execution

Fully visualize your entire data pipeline

Join Operations Imbedded Hive Flows

Functions, Filters and Groupings

ResultsSource SYNC

Confidential 10Confidential & Proprietary

PERFORMANCE MONITORING & VISIBILITY

Pinpoint bottlenecks and identify causes

EXECUTING WAITING

Detailed Performance Stats

See how all apps consume resources as they run

Confidential 11Confidential & Proprietary

PERFORMANCE MONITORING & VISIBILITY

Pinpoint bottlenecks and identify causes See how all apps consume resources as they run

Confidential 12Confidential & Proprietary

PERFORMANCE MONITORING & VISIBILITY

View Application and Unit of Work Details Surface Important Metrics and Information

See roll-up metrics on applications

Support monitoring Hive Server applications

See the status of each Unit of Work including HQL statements. Drill down to

get UoW details.

Confidential 13Confidential & Proprietary

PERFORMANCE MONITORING & VISIBILITY

View Application and Unit of Work Details Surface Important Metrics and Information

Counters Properties Environment

Confidential 14Confidential & Proprietary

ACHIEVING OPERATIONAL EXCELLENCE

• Performance monitoring and visibility for all big data applications- See everything in one place with a single view of all data applications real-time performance

metrics across the entire enterprise• Instantly understand the business context of a problem

- Knowing you have a problem is one thing but understanding the end to end & downstream dependencies, the quality and service guarantees, who is impacted, the owner(s) and the business criticality of the failure is game changing

• Easily collaborate across teams to resolve issues faster- Multi-tenant architectures are inherently chaotic. Accelerate issue resolution by delivering the right

information to the appropriate individuals and teams so everyone is informed and can work together to resolution

• Manage the business of big data- Get the operational visibility and controls you need to effectively manage priorities, policies,

capacity, compliance and utilization.

Balancing a fully operational data processing operation and customer demands is challenging

1

2

3

4

Confidential 15Confidential & Proprietary

UNDERSTANDING THE CONTEXT OF THE PROBLEM

Quickly identify the operational and business context

Quickly and easily identify execution errors

Not all problems are created equal

App Name Owner Organization Cluster A or B Privacy Level Production or Dev Custom Tags More …

Confidential 16Confidential & Proprietary

UNDERSTANDING THE CONTEXT OF THE PROBLEM

Quickly identify the operational and business context

Quickly and easily identify execution errors

Confidential 17Confidential & Proprietary

ACHIEVING OPERATIONAL EXCELLENCE

• Performance monitoring and visibility for all big data applications- See everything in one place with a single view of all data applications real-time performance

metrics across the entire enterprise• Instantly understand the business context of a problem

- Knowing you have a problem is one thing but understanding the end to end & downstream dependencies, the quality and service guarantees, who is impacted, the owner(s) and the business criticality of the failure is game changing

• Easily collaborate across teams to resolve issues faster- Multi-tenant architectures are inherently chaotic. Accelerate issue resolution by delivering the right

information to the appropriate individuals and teams so everyone is informed and can work together to resolution

• Manage the business of big data- Get the operational visibility and controls you need to effectively manage priorities, policies,

capacity, compliance and utilization.

Balancing a fully operational data processing operation and customer demands is challenging

1

2

3

4

Confidential 18Confidential & Proprietary

NURTURE A CULTURE OF OPERATIONAL EXCELLENCE

Ensure that business, development, IT operations can collaborate seamlessly when it matters

Confidential 19Confidential & Proprietary

COLLABORATIONShare and set alerts with any custom views and analytics

Share privately with an existing team or publically for all users to reference.

Confidential 20Confidential & Proprietary

COLLABORATIONCreate JIRA issues with views and data for quickly

collaborating to resolve performance problemsIntegrate alerts with popular notification platforms

like HipChat, PagerDuty, & Nagios

With one click, create a Jira issue with a link to this view

Confidential 21Confidential & Proprietary

COLLABORATIONCreate JIRA issues with views and data for quickly

collaborating to resolve performance problemsIntegrate alerts with popular notification platforms

like HipChat, PagerDuty, & Nagios

Automatically send app status notifications via webhooks or JMX

Confidential 22Confidential & Proprietary

ACHIEVING OPERATIONAL EXCELLENCE

• Performance monitoring and visibility for all big data applications- See everything in one place with a single view of all data applications real-time performance

metrics across the entire enterprise• Instantly understand the business context of a problem

- Knowing you have a problem is one thing but understanding the end to end & downstream dependencies, the quality and service guarantees, who is impacted, the owner(s) and the business criticality of the failure is game changing

• Easily collaborate across teams to resolve issues faster- Multi-tenant architectures are inherently chaotic. Accelerate issue resolution by delivering the right

information to the appropriate individuals and teams so everyone is informed and can work together to resolution

• Manage the business of big data- Get the operational visibility and controls you need to effectively manage priorities, policies,

capacity, compliance and utilization.

Balancing a fully operational data processing operation and customer demands is challenging

1

2

3

4

Confidential 23Confidential & Proprietary

MANAGING THE BUSINESS OF BIG DATA

Powerful Search& Custom Views

Monitor and analyze business critical data processes

Easily detect apps that violate SLA’s and policies

Views can be just for you, for a team, public or private, and categorized under general status or application specific

Quickly find and filter what you are looking for and save as a custom view

`

Confidential 24Confidential & Proprietary

MANAGING THE BUSINESS OF BIG DATA

Powerful Search& Custom Views

Monitor and analyze business critical data processes

Easily detect apps that violate SLA’s and policies

Metrics for all apps on the production cluster that failed to execute in under 30 minutes…

…or all apps that use more than their allotment of mappers

Confidential 25Confidential & Proprietary

MANAGING THE BUSINESS OF BIG DATA

Powerful Search& Custom Views

Monitor and analyze business critical data processes

Easily detect apps that violate SLA’s and policies

Measure and compare key analytics and metrics of application performance trends over

any period of time…

…Set SLA thresholds, observe and detect performance anomalies

Confidential 26Confidential & Proprietary

MANAGING THE BUSINESS OF BIG DATA

Segment performance by team, by department or custom tags for role-based views, chargeback models, and capacity planning

Reporting & Planning

Marketing

Sales

Compliance

Data science team

QA cluster

Production cluster

Performance of all apps owned by the Data Scientist team

Confidential 27Confidential & Proprietary

MANAGING THE BUSINESS OF BIG DATA

Segment performance by team, by department or custom tags for role-based views, chargeback models, and capacity planning

Reporting & Planning

Deeper analytics

Compliance

Chargeback reporting

Capacity planning

Any custom search filter can be saved and exported for reporting, to generate an audit

artifact, or to drop into your favorite spreadsheet

Confidential 28Confidential & Proprietary

MANAGING THE BUSINESS OF BIG DATAVisualize Lineage - See exactly how each app ingests,

manipulates and outputs dataFurther inspect lineage by detecting apps that write to,

or read from, a given dataset

Confidential 29Confidential & Proprietary

MANAGING THE BUSINESS OF BIG DATAVisualize Lineage - See exactly how each app ingests,

manipulates and outputs dataFurther inspect lineage by detecting apps that write to,

or read from, a given dataset

Show all apps that interact with the dataset in “rain.txt”

Confidential 30Confidential & Proprietary

MORE INFORMATION…

Visit us at www.driven.io