Open-BDA Hadoop Summit 2014 - Mr. Krish Krishnan (Driving Business Value – Big Data Platforms)

Driving Business Value – Big Data Platforms

Hadoop Summit

Karachi, Pakistan

Nov 18th 2014

Content and services via connected products Page 4 Hortonworks © 2014

New routes to market via intelligent objects

Everything

has a URL

Remote sensing of objects and environment

Augmented reality

Building and infrastructure management Cameras and

microphones widely deployed

Over 50% of Internet connections are things:

2011: 15+ billion permanent, 50+ billion intermittent

2020: 30+ billion permanent, >200 billion intermittent

Situational decision support Source: Gartner Keynote at Hadoop Summit 2013

Emerging Trends 2010-2015

Page 1Nick Jones and Alexander Linden26A, ESC17, 11/05, AE

© 2005 Gartner, Inc. and/or its affiliates. All rights reserved. Reproduction of this publication in any form without prior written

permission is forbidden. The information contained herein has been obtained from sources believed to be reliable. Gartner disclaims

all warranties as to the accuracy, completeness or adequacy of such information. Gartner shall have no liability for errors, omissions

or inadequacies in the information contained herein or for interpretations thereof. The reader assumes sole responsibility for the

selection of these materials to achieve its intended results. The opinions expressed herein are subject to change without notice.

Disruption: It's Not Just About Technologies

Identity/AccessManagement

Speech Recognition

Computer-BrainInterface

TruthVerification

ObjectIdentification

Linux

Wikis

Grid ComputingTablet PC

InformationExtraction

Location-AwareServices

Semantic Web

Service-OrientedArchitecture

UnifiedCommunications

Ultra Wide Band

RFID

SocialNetworkAnalysis

Web-Services-Enabled BusinessModels

Electronic Ink/Digital Paper

WebServices

InstantMessaging

VoIP

Sensor Networks

Smartphone

Really SimpleSyndication

Augmented Reality

4G Wireless

Blogging

Podcasting

IP Television

LocationSensing

Design Innovation

ProactiveTransparency

PersonalizedPricing

Counterfeit Reality

CollectiveIntelligence Perfect Recall

Real-Time Enterprise

UbiquitousAccess

Smart Objectsand AmbientIntelligence

Privacy Redefined

Global Sourcing

BusinessProcessManagement

Voice/DataConvergence

Microcommerce

Self-Sufficiency

GreenfieldBusiness

Global Micro-Business

Seamless Serviceto Self-service

Feedback Society

SemanticConnectivity

EmergingTechnologies

EmergingCapabilities

EmergingBusiness Models

Innovation is no longer the province of a few leading-edge companies — it is the lifeblood required for the

survival of any enterprise. Business and technology planners must understand how to harness the disruptive

potential of technology advances through a realistic assessment of emerging capabilities and IT-enabled

trends, and their impact on business and society. Planners must also acknowledge that disruptions arise not

just from new technology, but also from new applications or convergence in existing technologies that

drive new capabilities and new business models.

This emerging trends presentation identifies the most disruptive trends arising from emerging information

technology, and assesses their potential impact on the workforce, consumers, business, government and

society in the five to 10-year time frame.

@copyright Sixth Sense Advisors 2014 3

From Transactional to Behavioral

@copyright Sixth Sense Advisors 2014

The challenge facing the business today is the ability to influence the buyer decisions in a

window of opportunity that does not last long. The analytics available at a personalization

level drives the buyer whether it is choosing a Doctor or buying a Donut.

To compete in this new era, businesses need to be driven by data and analytics, which

are largely different from traditional transactions and campaigns

The “Gen Z” and “Millinieal Generation” of buyers will not be swayed by traditional

engagement models of selling products and services

4

A Growing Trend


Executive Ask

• Why is our competition performing better in the same markets as us?

• What is the real revenue impact to the organization?

• Why do our campaigns not predict the accurate revenue lift?

• How can we improve our brand and align to customer sentiment?

• Who can help us get to the bottom of the data abyss?

• When do we get the real insights from Analytics?


Forces Shaping Business


Business Evolution – Product to Customer Focus Shifts


Image source - internet

8

Decision Support – Now & Then


Customer Promotion

Social Presence

Behavior Analytics

Competition Market

Value

Transactions

Price Product Channel

Sales

Promotion

Market

Price

Transactions

Customers

9

Innovation as Way of Life


Old Brick

New Brick

10

State of Data


Big Data

• Data that cannot be processed using traditional data processing techniques due to size, scale and formats

• Its beyond just that

– Complexity

– Ambiguity

– Veracity


Noise vs Value


Is Big Data Noise or Value?

Is Big Data a passing hype?

Is there real value behind Big Data?

Is there any measurable and actionable

insight from Big Data?

Is there a need to invest in this now?

Image source - internet

13


Example


Tweet: @jdoe – very disappointed with @united @checkin lousy svc,

bad mgmt, long lines #fail.

20000 retweets. 4 hours ago

IT Perspective Text of 140 chars will be stored as string. The data model for this will be a table with source, content, datetime.

Business Perspective User – JDoe Brand – United Sentiment – Negative Process – Check-in Time – Waiting in long lines Impact – shared 20000 times in 4 hours

15

Why Does Big Data Differ


IT does not know data

Business knows the intelligence to be applied to the data to derive value

Business knows how to discover data patterns (manual and automated)

Business understands the semantics better

Business can perform data interrogation in an experiment and associate rules of engagement early on for data usefulness

Business can sift the data to curate the context

Big Data needs to be curated to be useful

16

@copyright Sixth Sense Advisors 2014 17 Source: McKinsey Presentation on Marketing

Big Data Challenges


Complex Ambiguous Formats /

Availability

Minimally Organized

Complex Data Quality Issues

Metadata / Semantics

Needs Intervention

Needs Discovery /

Contextualization Needs Analysis

18

Perspective


Data

Insights

19

The Paradigm Shift


IT

• Facilitate

• Maintain

• Support

• Manage

Business

• Driver

• Budget Sponsors

• Program Owner

• Define & Consume

IT

• Driver

• Program Owner

• Budget Sponsor

• Maintain

• Support

Business

• User

Data Warehouse Big Data

20

Analysis


Processing – The Details


Tag

Categorize

Classify

Cleanse (Data Quality Rules)

Semantic Integration

Measure

Visualize

22

Big Data Platforms

Future Visualization


Big Data Architecture


Big Data Sources

Hadoop (and / or NoSQL)

Traditional Data

Sources

ETL ELT CDC

Staging Or ODS

ETL ELT EDW

BI Analytics

Discovery

Data Mining Algorithms Acquire

Big Data Analytics

24


Infrastructure In

fras

tru

ctu

re

Ch

alle

nge

s &

So

luti

on

s

Challenges To Address

Semantic Data Integration

Compression & Storage

High Capacity Warehouse

Security and Governance

Scripting and Development Tools

Complex Event Processing

Solutions Available Today

Columnar Databases

Workload Optimization

Analytic Accelerators

Hadoop / Map Reduce

No SQL Engines

Stream Computing


Big Data Analytics


Existing EDW

Machine Learning

Text Mining Analytics

Coding & Learning

Semantic Knowledge Base Metadata & Semantic

Layer

27

28

Dealing with Big “Data Problems”

Data silos in disjointed systems

Multiple data sources - overlapping, conflicting

Timely processing of large volumes of data

Partial, inaccurate, inconsistent.. data

28

Single Data Platform

Reports

Analytical DBMS

Analytics Cluster

Data Asset Catalog

Analytical DBMS

Dashboards

Data Discovery

Interactive Queries

Batch Queries

Web Applications

Activity Logs

NoSQL

Reference Data

Device Apps

Probes

3rd Party

Device

User Profile

POI, Map

Activity Sensor

Dat

a In

take

ETL,

dat

a cr

un

chin

g,

attr

ibu

tio

n, M

L A

lgo

rith

ms

Agg

rega

tio

n

HDFS

Analytical DBMS

Big Data Analytics Platform Data Flows

29 @copyright Sixth Sense Advisors 2014 29

Analytics

• Behavioral Analytics

• Cohort Analytics

• Collections Analytics

• Competitive Analytics

• Financial Services Analytics

• Fraud Analytics

• Marketing Analytics

• Pricing Analytics

• Sales Analytics

• Risk Analytics

• Supply Chain Analytics

• Talent Analytics

• Channel Analytics

• Logistics Analytics


Trends 2015

• Business Users Drive Big Data Initiatives

• Analytics is the “new” center stage

• Data Discovery and Storyboard Patterns Emerge Stronger

• Hadoop-Based Data Lakes / Swamps Unite with Data Warehouses

• Predictive Analytics Lend Fresh Insight From Big Data Explorations

• Prescriptive Analytics Bring New Insight to Current Business Processes


Measuring Value

Increasing Revenue /

Opportunity / Market

Lowering Costs / Risks

/ Maintenance

Value


Critical Success Factors


Business needs to own and execute the Big Data program

Data collection and discovery is the most critical step

Metadata is needed to process the data prior and post Data Warehouse

integration

Data quality can be processed by integrating Taxonomies

Data visualization is needed to discover data

Metrics and Metadata will be the bridge to integrate to the Data Warehouse

33

Questions


Thank You

Contact

[email protected]

Twitter: @datagenius

Images Source - Internet


mailto:[email protected]

mailto:[email protected]

References

• Gartner

• McKinsey

• MIT Innovation Labs

• Google

• Facebook

• LinkedIn
