ACHIEVE OPERATIONAL EXCELLENCE WITH

BIG DATA APPS

Confidential

WHY NOW?

2

As big data applications become the engine of your data management strategy, they must meet higher standards of quality, reliability, and manageability.

Confidential

5 BEST PRACTICES TO ACHIEVE OPERATIONAL READINESS

3

Balancing a fully operational data processing cluster and customer demands is challenging.

4

3

2

1Performance monitoring and visibility for big data apps• Increase the quality and efficiency of your clusters with a single integrated view of your data

applications and real-time performance metrics across all environments.

Identifying and tuning poorly performing applications• Spend less time wading through Hadoop logs, ResourceManager and source code to find

issues with your data pipelines. Instead, use that time optimizing your environment.

SLA adherance• As our Hadoop clusters grow, managing our applications and continuing to meet SLAs

becomes increasingly complex. Answer these questions in seconds.

Collaboration across teams to resolve issues faster• Collaboration between all roles that interact with an application, data scientists, developers

and operations, the quality and efficiency of your application increases.

Utilize existing project management processes• Identify and resolve issues quickly by integrating your existing tools and services, such as

Jira, Hipchat, Slack etc into your Hadoop environment.5

Confidential

PERFORMANCE MONITORING & VISIBILITY

4

Pinpoint bottlenecks and identify causes

See all successful, failed, pending processes…

…quickly segment your aggregate view by app name, team , cluster etc…

Comprehensive view of all your data processing execution Fully visualize your entire data pipeline

Immediately understand the status of all your data applications

Confidential

PERFORMANCE MONITORING & VISIBILITY

5

Fully visualize your entire data pipelineComprehensive view of all your data processing executions

SOURCE RESULTSSYNC JOIN OPERATIONSIMBEDDED HIVE FLOWS

FUNCTIONS, FILTERS AND GROUPINGS

Confidential

PERFORMANCE MONITORING & VISIBILITY

6

Seamlessly monitor heterogeneous environments

Confidential

5 BEST PRACTICES TO ACHIEVE OPERATIONAL READINESS

7

Balancing a fully operational data processing cluster and customer demands is challenging.

2

Performance monitoring and visibility for big data apps• Increase the quality and efficiency of your clusters with a single integrated view of your data

applications and real-time performance metrics across all environments.

Identifying and tuning poorly performing applications• Spend less time wading through Hadoop logs, ResourceManager and source code to find

issues with your data pipelines. Instead, use that time optimizing your environment.

SLA adherance• As our Hadoop clusters grow, managing our applications and continuing to meet SLAs

becomes increasingly complex. Answer these questions in seconds.

Collaborate across teams to resolve issues faster• Collaboration between all roles that interact with an application, data scientists, developers

and operations, the quality and efficiency of your application increases.

Utilize existing project management processes• Identify and resolve issues quickly by integrating your existing tools and services, such as

Jira, Hipchat, Slack etc into your Hadoop environment.

1

3

4

5

Confidential8

UNDERSTAND THE CONTEXT OF THE PROBLEMQuickly and easily identify execution errorsQuickly identify all failing applications

App NameOwnerOrganizationCluster A or BPrivacy LevelProduction or DevCustom TagsMore …

Not all problems are created equal

Confidential9

UNDERSTAND THE CONTEXT OF THE PROBLEMQuickly identify the operational and business context Quickly and easily identify execution errors

Confidential

PERFORMANCE MONITORING & VISIBILITY

10

Pinpoint bottlenecks and identify causes

EXECUTING WAITING

Pinpoint bottlenecks and identify causes

DETAILED PERFORMANCE STATS

Confidential

5 BEST PRACTICES TO ACHIEVE OPERATIONAL READINESS

11

Balancing a fully operational data processing cluster and customer demands is challenging.

3

Performance monitoring and visibility for big data apps• Increase the quality and efficiency of your clusters with a single integrated view of your data

applications and real-time performance metrics across all environments.

Identifying and tuning poorly performing applications• Spend less time wading through Hadoop logs, ResourceManager and source code to find

issues with your data pipelines. Instead, use that time optimizing your environment.

SLA adherance• As our Hadoop clusters grow, managing our applications and continuing to meet SLAs

becomes increasingly complex. Answer these questions in seconds.

Collaborate across teams to resolve issues faster• Collaboration between all roles that interact with an application, data scientists, developers

and operations, the quality and efficiency of your application increases.

Utilize existing project management processes• Identify and resolve issues quickly by integrating your existing tools and services, such as

Jira, Hipchat, Slack etc into your Hadoop environment.

1

2

4

5

Confidential

SLA ADHERENCE

12

Pinpoint bottlenecks and identify causes

Powerful search & Custom views

Quickly find and filter what you are looking for and save as a custom view

Views can be just for you, for a team, public or private, and categorized under general status or application specific

Confidential

SLA ADHERENCE

13

Further inspect lineage by detecting apps that write to, or read from, a given dataset

Show all apps that interact with the dataset in “rain.txt”

Confidential

5 BEST PRACTICES TO ACHIEVE OPERATIONAL READINESS

14

Balancing a fully operational data processing cluster and customer demands is challenging.

4

Performance monitoring and visibility for big data apps• Increase the quality and efficiency of your clusters with a single integrated view of your data

applications and real-time performance metrics across all environments.

Identifying and tuning poorly performing applications• Spend less time wading through Hadoop logs, ResourceManager and source code to find

issues with your data pipelines. Instead, use that time optimizing your environment.

SLA adherance• As our Hadoop clusters grow, managing our applications and continuing to meet SLAs

becomes increasingly complex. Answer these questions in seconds.

Collaborate across teams to resolve issues faster• Collaboration between all roles that interact with an application, data scientists, developers

and operations, the quality and efficiency of your application increases.

Utilize existing project management processes• Identify and resolve issues quickly by integrating your existing tools and services, such as

Jira, Hipchat, Slack etc into your Hadoop environment.

1

2

3

5

Confidential

NURTURE A CULTURE OF OPERATIONAL EXCELLENCE

15

Ensure that business, development, IT operations can collaborate seamlessly when it matters

Confidential

5 BEST PRACTICES TO ACHIEVE OPERATIONAL READINESS

16

Operate as a team

Associate users with teams

Confidential

COLLABORATION

17

Share and set alerts with any custom view and analytics.

Share privately with an existing team or publically for all users to reference.

Confidential

5 BEST PRACTICES TO ACHIEVE OPERATIONAL READINESS

18

Balancing a fully operational data processing cluster and customer demands is challenging.

Performance monitoring and visibility for big data apps• Increase the quality and efficiency of your clusters with a single integrated view of your data

applications and real-time performance metrics across all environments.

Identifying and tuning poorly performing applications• Spend less time wading through Hadoop logs, ResourceManager and source code to find

issues with your data pipelines. Instead, use that time optimizing your environment.

SLA adherance• As our Hadoop clusters grow, managing our applications and continuing to meet SLAs

becomes increasingly complex. Answer these questions in seconds.

Collaborate across teams to resolve issues faster• Collaboration between all roles that interact with an application, data scientists, developers

and operations, the quality and efficiency of your application increases.

Utilize existing project management processes• Identify and resolve issues quickly by integrating your existing tools and services, such as

Jira, Hipchat, Slack etc into your Hadoop environment.

1

2

3

4

5

Confidential

EXISTING PROCESSES

19

Create JIRA issues with views and data for quickly collaborating to resolve performance problems

Integrate alerts with popular notification platforms like HipChat, PagerDuty, & Nagios

Automatically send app status notifications via webhooks or JMX

Confidential

COLLABORATION

20

Create JIRA issues with views and data for quickly collaborating to resolve performance problems

With one click, create a Jira issue with a link to this view

Confidential

5 BEST PRACTICES TO ACHIEVE OPERATIONAL READINESS

21

Balancing a fully operational data processing cluster and customer demands is challenging.

4

3

2

1Performance monitoring and visibility for big data apps• Increase the quality and efficiency of your clusters with a single integrated view of your data

applications and real-time performance metrics across all environments.

Identifying and tuning poorly performing applications• Spend less time wading through Hadoop logs, ResourceManager and source code to find

issues with your data pipelines. Instead, use that time optimizing your environment.

SLA adherance• As our Hadoop clusters grow, managing our applications and continuing to meet SLAs

becomes increasingly complex. Answer these questions in seconds.

Collaboration across teams to resolve issues faster• Collaboration between all roles that interact with an application, data scientists, developers

and operations, the quality and efficiency of your application increases.

Utilize existing project management processes• Identify and resolve issues quickly by integrating your existing tools and services, such as

Jira, Hipchat, Slack etc into your Hadoop environment.5

Confidential

TAKE THE DRIVEN TOUR

For a walk-through of all the features of Driven,

Go to our Showcase interactive demo

http://showcase.cascading.io/index.html#/login

THANK YOU

APPENDIX

Confidential

End-to-end operational telemetry metadata for big data applicationsAccessible via Web browser, command-line interface (CLI), or simple search queriesEasy integrations through JMX and upcoming Driven SDK

… THROUGH A SCALABLE, SEARCHABLE METADATA STORE

Telemetry metadata(SSL)

YARN

HADOOP APPS AND INFRASTRUCTURE

APPLICATIONS

Plugin

25

HADOOP CLUSTERS

WAR

files Web App

Server

Server

Web CLI JMX

Web AppServer

SCALE OUT

SCALE OUT

Confidential

Commercial

Training Consulting

Community

Free community support through our mailing list and our online forumswww.cascading.org/support | forums.cascading.io

We offer short-term consulting engagements designed to help customers with mentoring and best practices

Developer training for Cascading and Scalding,

private training also availablewww.cascading.io/services/training/

Varying levels of technical support are available to support your production deploymentswww.cascading.io/services/support

Supporting our Customers & Community

SUPPORT OPTIONS

26