Big Data & Real-time Campaign GSE - External€¦ · • Additional “sessionization”and aggregation analysis help to detect valuable click patterns and provide insights for: –

© 2011 IBM Corporation

Telecom Solutions Lab

BAO & Big Data OverviewApplied to Real-time Campaign

GSE

Joel VialeTelecom Solutions Lab – Solution Architect



Agenda

2

• BAO & Big Data - Overview

• Customer use-cases

• Live Prototypes:

• Streams for Real-time Campaign

• Big Insights for Web Log Analysis



3

The Business Analytics Journey: Transform data into actionable insight!

AnalyzeIntegrate

Transactional& Collaborative Applications

Manage

Business Analytics Applications

Internal & External Information Sources

Cubes

Streams

Big Data Master Data

Content

Data

Streaming

Information

Data Warehouses

Govern

QualitySecurity & PrivacyLifecycle



4

What is Big Data? How could it impact your organization?

Big Data technologies describe a new

generation of technologies and architectures, designed to economically extract value from very large volumes of a wide variety of data, by enabling velocity capture, discovery and/or analysis.

- Matt Eastwood, IDChttp://www.tweetdeck.com/twitter/matteastwood/~hHgsU



5

The Big Data Challenge

• Manage and benefit from massive and growing amounts of data

• Handle varied data formats (structured, unstructured, semi-structured) and increased data velocity

• Exploit BIG Data in a timely and cost effective fashion

Collect Manage

Integrate Analyze

COLLECT MANAGE

INTEGRATE ANALYZE



6

Bring Together a Large Volume and Variety of Data to Find New Insights

Identify criminals and threats from

disparate video, audio, and data

feeds

Make risk decisions based on real-

time transactional data

Predict weather patterns to plan

optimal wind turbine usage, and

optimize capital expenditure on

asset placement

Detect life-threatening

conditions at hospitals in time

to intervene

Leverage Multi-channel customer

sentiment and experience

analysis to upsell in real time



7

The Solution – IBM’s Big Data PlatformBring together any data source, at any velocity, to generate insight

� Analyzing a variety of data at enormous volumes

� Insights on streaming data

� Large volume structured data analysis

IBM Big Data Platform

• Variety

• Velocity

• Volume



8

2 ways of creating insights: Streaming and Storing Big Data

• Real time analysis of data-in-motion

• Structured or unstructured

• Analytic operations on streaming data in real-time

Data QueriesResults

b) streaming data

Data QueriesResults

b) streaming data

Data QueriesResultsData QueriesResults

b) streaming data

• Managing and analyzing Internet-scale volumes

• Structured or unstructured data

• Based on Google’s MapReduce technology

• Inspired by Apache Hadoop; compatible with its ecosystem and distribution

• Well-suited to batch-oriented, read-intensive applications

• Integrated Text Analytics



9

9

-Distributed File System (HDFS)

-Map/Reduce

How to deal with Big Data: Split a big job into smaller piecesHighly Massive Parallel Processing (MPP)



10

InfoSphere Streams enables highly scalable stream processing

� InfoSphere Streams provides – a programming model for defining data flow graphs consisting of data sources

(inputs), operators, and sinks (outputs)– controls for fusing operators into processing elements (PEs)– infrastructure to support the composition of scalable stream processing applications from these components

– deployment and operation of these applications across distributed x86 processing nodes,when scaled-up processing is required



11

Big Data complements traditional warehouse & analytics infrastructure

Streams

Internet

Scale

TraditionalWarehouse

In-Motion

Analytics

Data Analytics,

Data Operations

& Model

Building

Results

Internet Scale

Database &

Warehouse

At-Rest Data

Analytics

Results

Ultra Low

Latency Results

InfoSphere Big

Insights

InfoSphere Big

Insights

Moving Beyond the Traditional Warehouse

Traditional /

Relational

Data Sources

Traditional /

Relational

Data Sources

Non-Traditional /

Non-Relational

Data Sources

Non-Traditional /

Non-Relational

Data Sources

Non-Traditional/

Non-Relational

Data Sources

Non-Traditional/

Non-Relational

Data Sources

Traditional/Relatio

nal Data Sources

Traditional/Relatio

nal Data Sources



Agenda

12








13

Examples of Streams use-cases

Stock market• Impact of weather on securities prices

• Analyze market data at ultra-low latencies

Fraud prevention• Detecting multi-party fraud• Real time fraud prevention

e-Science• Space weather prediction• Detection of transient events• Synchrotron atomic research

Health & Life Sciences• Neonatal ICU monitoring• Epidemic early warning system• Remote healthcare monitoring

Transportation• Intelligent traffic management

Law Enforcement, Defense &

Cyber Security• Real-time multimodal surveillance• Situational awareness• Cyber security detection

Manufacturing• Process control for microchip fabrication

Natural Systems• Wildfire management• Water management

Telephony• CDR processing• Social analysis• Churn prediction• Geomapping

Other• Smart Grid• Text Analysis• Who’s Talking to Whom?• ERP for Commodities• FPGA Acceleration



Streams in Telecommunications

14

Data in motion

CDRs

Billing

CRM

Location

Account Mgt

Internet

Network

Millions of

events per

second

Microsecond

Latency

Dropped Calls

Outgoing International CallsCall Duration

Extra Call

Contract Expiration

Entered new cell

New Top-Up

5 minutes left on pre-paid

MDM

EDW

Invoice Issued

Campaign Mgt Real-time

Promo

Fraud Detection

Service Assurance Network

Monitoring

Real-time Monitoring (Voucher

Recharge & Service Usage)

3 Dropped Calls in the last

hour

Location-based Promo

5 Outgoing international Calls in the last

day

100 SMSs sent in 1 hour500$ top-up on pre-paid

No game download in last 30 minutes

500 Failed SMS Deliveries last 10 minutes

Aggregated Data Records at Service

Level

Aggregated Data Records at Cell

Level

Aggregated Voucher Recharge

Data Aggregated at

Single Customer

Level

Data Aggregated at

Cell or Service

Level



CDR cleansing and pre-processing

The Pain Point:

• A CSP has 6 Billions CDRs per day!

• As much as 500,000 per second peak rate

• Kept up, but re-processing meant waiting for nights/weekends to catch up

The Solution:

• InfoSphere Streams to clean and pre-process CDRs

• De-duplication of CDR against 15 days of data (90B CDRs)

• Offloading CDRs processing to Streams platform increased the performance of their warehouse for other analytics

• Single platform for mediation and real time analytics reduced IT complexity



Real-time marketing campaign

InsightInsight

InformationInformation

prescriptive

prescriptive

DataData active

active

Business flexibility & responsiveness

Business value

The Pain Point

� 100M CDRs per day from SMS from 25M subscribers, only used to send bills to customers

The Solution

� InfoSphere Streams to create real-time marketing promotions and monetize the CDRs



Typical Telco Use-Cases with InfoSphere Streams

� ETL-like for massive data to off-load traditional ETL and Warehouse

� Mediations

� Prediction and prevention of customer churn– In-motion analytics can help identify ‘group leaders’

� Real Time context sensitive promotions– Location based promotions– Customer minutes usage based promotions– Customer smart phone browsing pattern based promotions– Network equipment utilization based promotions

� Real Time & pro-active network monitoring

� Real Time call traffic & revenue monitoring

� SMS spam filtering

� Fraud detection17



Typical Telco Use-Cases with InfoSphere Big Insights

CDR Use Cases:

� User behavior analysis

� Prediction on the basis of Customer churn

� Service association analysis

Social Media Analytics: Get insights, improve campaign, influence opinions

Fraud Detection: Monitoring/Mining transaction logs to detect fraud activities

Recommendation Engines: Improving user experience and likelihood of purchase

Network Traffic Logging: Network optimization & fault detection

Advertising Optimization: in support of advertising based models

Server Logs: Fault detection / Performance related analysis

Email and Email Logs:

� Consumer email analysis & decision system

18



Product Capabilities

Big Insights at the core of Cognos Consumer Insight

BLOGSBLOGS

DISCUSSION FORUMSDISCUSSION FORUMS

TWITTERTWITTER

NEWSGROUPSNEWSGROUPS

FACEBOOKFACEBOOK

Source Areas

� Dimensional Analysis

� Filtering

� Voice

� Keyword Search

� Dimensional Navigation

� Drill Through to Content

� Relevant Topics

� Associated Themes

� Ranking and Volume

� Relationship Tables

� Relationship Matrix

� Relationship Graph

COMPREHENSIVE ANALYSIS

SENTIMENT

EVOLVING TOPICSAFFINITY ANALYTICS

Business Drivers

Customer CareCustomer CareCorporate ReputationCorporate Reputation

Campaign EffectivenessCampaign Effectiveness

Competitive Analysis

Product Insight

MULTILINGUALMULTILINGUAL

19



Agenda

20








Enterprise Marketing Management

Advanced BI, Reporting & Real-time

Monitoring

Real-time Analytics of

Data-in-Motion

Real-time Campaign – Solution Architecture

Advance

d

Predictiv

e

Analytics

High-Performance

Data Warehouse

Appliance

InfoSphere Streams

Unica Suite

SPSS

Cognos Suite

Netezza



Outbound campaign, enhanced with Real-time Monitoring and advanced predictive analytics

Events Collaboration Content MonitoringAnalytics Rules

Product Manager

Marketing Manager

VP of Marketin

g

Customer

Define Marketing Strategy

Create Campaig

n

Identify Target

Customers and

Offering

Approve Campaig

n

Monitor & Evaluate Campaig

n

Execute Campaig

n

Delivery Channel

Real-time Monitoring

Detect situations and patterns in service usage. Determine the need of a campaign

Predictive Analytics

Segment Customers based on Churn Prediction, likelihood of campaign response, LTCV, etcO

Presence & Rules based channel

selection

Send campaign message on appropriate channel (including social networks), based on customer preferences and availability/presence

Monitor key KPIs

• Sales/Usage increase• % of campaign response

22

Collect and

aggregate xDRs

Network

Real-time xDRs

Processing

Collect, correlate and aggregate data records in real-time



23

Real-time campaign, based on network events

Manage Campaign Delivery

Monitor Campaign Execution

Event Based Campaign Processing

Evaluate Campaign Effectiveness

Real-time Event Detection

InfoSphere Streams correlate multiple data records in real-time down to the customer level.Here: detects a number of dropped calls in a certain time

Event-based Campaign

The event-based campaign processing is triggered: a campaign is executed in Unica Campaign.

Message Delivery

The campaign message is delivered to the customer on the appropriate channel

Campaign monitoring

Monitor campaign execution and analyze effectiveness

CDRs

CDR

Feed

Dropped Calls Filtering

Dropped Calls Aggregation

Per Subscriber

InfoSphere StreamsReal-time event detection



Agenda

24








The Sample Outdoors Company has many visitors a year to their

website and only a subset resulted in purchases. They would like to know what is

happening to the other visits?

25



Every time the user browses a page, the click-stream data is logged

User goes to web site

Add item

to cart

Search for

item

Userchecks

out

26



• Web logs capture click streams of user

• Basic analysis of web logs can provide valuable insights to web site usage and user behaviors

– How many users going to my web site?

– What pages and products are people looking at?

• Additional “sessionization” and aggregation analysis help to detect valuable click patterns and provide insights for:

– Improving customer service

– Ads promotion and sales improvement

– Detection of incidents and errors

Click Stream Data Analysis for Customer Insights

27



Big Insights Web Log Analysis Solution Overview

Web Server Logs

BigInsights Analytics

28



cluster

Solution Architecture for Web Log Analysis

JAQL

Aggregated reports on user

shopping behavior

O

Big Insights (HDFS, …)

Text Analytics (SystemT)

Raw Logs

29

Documents

Big Data & Real-time Campaign GSE - External€¦ · • Additional “sessionization”and aggregation analysis help to detect valuable click patterns and provide insights for: –