Real Time Big Data Framework

Sharath Hosagrahar

Lead Architect, Red Hat Global Partners & Alliances Organisation

8 October 2015

Source: http://blogs.ifsworld.com/2013/02/how-will-big-data-influence-your-finance-team/

Simple Definition

Big data is a collection of data from traditional and digital

sources inside and outside your company that represents a

source for ongoing discovery and analysis

TURN BIG DATA

INTO

ACTIONABLE

BUSINESS

INSIGHTS

• Collected from various sources

• Analyze and act on data separately

• Using past data to make strategic decisions

• Batch processing driven

BIG DATA AT REST

• Similar collection process as big data at rest

• Analytics occur in real time as the event happens

• Common in Telco e.g. analyzing mobile phone subscriber usage patterns

• Complex Event Processing with low latency

BIG DATA IN MOTION

• Real time data analysis and as a result, real time action

• Captured data can be stored in a big-data store for batch analysis

• Batch analysis to identify and model customer segments

• Real time analysis to act on an identified customer segment

• Combining batch and real time analysis provides maximum actionable business insights

COMBINING THE TWO

RED HAT REAL TIME BIG DATA FRAMEWORK

• Analyze large quantities of stream data in real time

• Scale out architecture

• A flexible analytical approach using JBoss BRMS

• Virtual integration with structured and unstructured data and connects to Internet of Things (IoT) devices

• Open Source & Standards Based

• Proven Success

HIGHLIGHTS Agile business insights and actions through real time analysis and integration of

data in motion and data at rest

• Real Time Marketing / Promotions

– Upsell/cross-sell

• Social media analysis

– Trends and News

• Location Based Services (LBS)

• Network Channel Analysis

ACTUAL CUSTOMER USE CASES

HIGH LEVEL SOLUTION ARCHITECTURE

• Distributed execution of Map-Reduce workloads based on data from various sources performed within JBoss Data Grid – Faster distributed processing in memory and within network – More complex processing scenarios can be achieved by combining

this workload with the decision management capabilities within JBoss BRMS

FEATURE 1: REAL TIME MAP REDUCE

Real-Time Processing Short Cycle Execution

Keyword extraction

Keyword extraction

Keyword extraction

Sum

Sum

Sum

Relay

receive

receive

receive morphological analysis

morphological analysis

morphological analysis

Watch the word ranking

Rule matching

• Modify decision rules in real time • Accumulate decision knowledge as rules and evolve

into an intelligent operational decision engine • Short Cycle Execution

FEATURE 2: REAL TIME DECISIONING

Real-Time Processing Short Cycle Execution

Keyword extraction

Keyword extraction

Keyword extraction

Sum

Sum

Sum

Relay

receive

receive

receive morphological analysis

morphological analysis

morphological analysis

Watch the word ranking

Rule matching

• Build virtual data views using JBoss Data Virtualization

– Integrate real time analysis of JBoss Data Grid with master information from RDBMS via JBoss Data Virtualization

– Analyst can view information via dashboard or existing business intelligence or custom applications

– Developers/Administrators can build virtual tables with an IDE-based tool or standard SQL notations

FEATURE 3: REAL TIME BUSINESS INSIGHTS

FEATURE 3: REAL TIME BUSINESS INSIGHTS

Data Virtualization

RDBMS

JBoss Data Virtualization

RDBMS

JBoss Data Grid

JBoss Data Grid

Deploy

Data Integration/ Virtualization

RED HAT JBOSS MIDDLEWARE COMPONENTS

RED HAT CONSULTING APPROACH

RED HAT METHODOLOGY

RED HAT CONSULTING SERVICES

CALL CENTER SYSTEM (AS-IS)

IVR

call

call

call

call

call

call

Tele Operator

End User

Call Log Call Log Call Log

Call Log

Call Log Call Log Call Log

Call Log

Call Log Call Log

Call Log Call Log

Call Log Call Log

Automatic

input

input

input

Write Program Hadoop

Batch Platform

Report

Manager

Analyze Report per 1 day

Call Center System

CALL CENTER SYSTEM (TO-BE)

IVR

call

call

call

call

call

call

Tele Operator

End User

Call Log Call Log Call Log

Call Log

Call Log Call Log Call Log

Call Log

Call Log Call Log

Call Log Call Log

Call Log Call Log

Automatic

input

input

input

Write Program Hadoop

Batch Platform

Report

Manager

Analyze Report per 1 day

Call Center System

JBoss Data Grid

JBoss BRMS

JBoss EAP

JBoss DV

Alert over threshold

Hive

Real Time Big Data Framework

Thank you

Red Hat Forum 2015

Energize Your Enterprise