Lambda Architecture 2.0 Convergence between Real-Time Analytics, Context-awareness and Online Learning

HUAWEI TECHNOLOGIES CO., LTD.

CEP & PME For Real-Time

Analytics

Big Data Technologies conference December 2014

Sabri SKHIRI – Head of the R&D Architecture Dpt., European Research Center

HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential

Agenda

1. Introduction

2. What is the Lambda Architecture & Lambda 2.0 Proposal with PME

3. Examples of Use cases with the PME

4. Conclusion


Sabri SKHIRI Head of R&D Architecture – Expert in Distributed system architecture, Soft Engineering & Data analytics

Scientific Publications (selected) A Distributed Data Mining Framework Accelerated with Graphics Processing Units. 2013 IEEE Conference on Cloud Computing and Big Data Large Graph Mining: Recent Developments, Challenges and Potential Solutions. 2012 Lecture Notes in Business Information Processing, Springer AROM: Processing Big Data with DataFlow Graphs and Functional Programming. 2012 IEEE conference on Cloud computing technology and science And 7 others.

Committer on Big Data open source projects launched @ EURA NOVA

R&D Projects at Huawei (selected)

Distributed Graph Storage & Traversal Graph Mining Libs (influence Mngt) Distributed in-memory Machine Learning Platform Pattern Matching Engine / CEP Stream Processing On Going R&D Projects Feature Engine (Deep Learning) Automatic Modeling Data PaaS

R&D Director & Co-founder of EURA NOVA BE, Private research company


Before Starting - The Big Data ambiguity Demystify the IT Vendor confusion about Big Data – “Is big data == Hadoop?”


Introduction Real-Time Analytics


IT Framework Based on Big Data

Data Value Openness

Leveraging Big Data in IT operations

Driving the Enterprise

operations through Data-

Driven / Knowledge-Driven

processes

Exploration on the big data ecosystem

Value chain from data sources to

insight consumers

Data-Driven Operation

Big Data Requirement Types of Telecom Carriers Main Directions


Real-time marketing Context-aware applications

Real-time context-aware QoS management

Intelligent Business Process Management (iBPM)

Trends analysis

Marketing automation

Intelligent Business Operations

Dynamic QoE Management

Proactive CEM

Network Management

Shift in Directions New Business models involve moving to Real-Time


Shift in Directions – Impact on the Technologies Moving from Batch to context-aware & Real-Time Reaction

Value of Data

Data Stored

Event

Info Delivered

Action

time

ETL ETL HD

FS

Hadoop

M/R

ETL ETL ETL

ETL ETL

DB

ETL ETL DWH

Acquire Organize Analyze

Analysi

s

applian

ce

Decide

Minutes/Hours/Days Sub-second

Detect & Decide

In-memory Analysis Accelerator

(On-line, incremental)

Complentary to Hadoop STACK

Context detection &

reaction

Data value over time

Systems must detect the “key”

information as they happen. The value

of the action decreases over time!

The Architecture shift

We are moving from a batch processing architecture to Real-

Time analysis and Context-aware infrastructure. The Hadoop-

based infrastructure is not well suited to this paradigm shift, but

can be used in a complementary way.

X

X


Matching Pattern among event Streams Going beyond traditional CEP

Matching complex patterns We can match complex definitions of situation including temporal relationships, event correlation and data from application (CRM, network management, HLR, etc.). We studied 20 Business cases and defined why traditional CEP engines cannot express them efficiently.


When should we use the Context? Leveraging the context-awareness

When we need to identify “interesting situations” If an application must be able to detect specific situations .Typical UC in Event-Based Marketing.

When we need to react to the context in RT If an application requires to answer to a specific context by taking an action, we need to define what is this context. The SmartSwitch can define such context. For instance: My flight is at 7:00PM, and it is delayed of 30m, and there is a Traffic Jam on the way from my location and the airport=> The service recommends me to leave my meeting in 10m. Typical UC in Marketing & Network.

When we need to correlate information from different systems or apps If we have to set up constraints on different events coming from different systems or applications without being able to touch at those systems, you can express all those constraints between events as correlations and even detecting all events that do no fit the pattern. Typical UC in Business Enablement Cube, Network and QoS.


Requirement Summary What is needed to support those new kind of RT Analytics Cases?

Expressing a context or a situation by mean of events We need to be able to express a situation by describing all the events that participate to this context and by defining the temporal relationships between them. Once we have defined this context definition (we call it pattern) we must be able to deploy this pattern at run-time, to listen all the involved events and to detect when this pattern occurs. This detection must done through different streams of millions of events per second.

What exist today to express and recognize those patterns? We have 3 existing technologies for in this area: 1. Event Stream Processing 2. Complex Event Processing (CEP) 3. Pattern Matching engine And integrating RT Contexts with

Predictive Models! We need to be able to associate a situation to a process of actions in which we can leverage all the batch-calculated models. But also … Correcting the model by a Feedback Loop Incrementally update the model


(Sybase)

Description and differences among those 3 Technos Pattern Matching is the only one that fits our requirements

Stream Processing (Usually) Distributed event processing infrastructure. Give the ability to define the Stream Processing as a DFG (aka Topology). No temporal support, no DSL (CQL), no support for Event correlation.

CEP Based on Stream processing Technology, the CEP exposes a language derived from SQL for event Processing. The objective is to compute KPIs over streams and to define patterns as Thresholds on those KPIs. The language is tuned for defining operations on Streams.

Pattern Matching Engine The objective is NOT to compute operations on streams but well to express complex temporal correlations between Event streams to define a context. The language is then not oriented to operations but well on expressing complex constraints.

[Tatbul 2006]

Storm + CEP Toolkit

http://esper.codehaus.org

Cayuga

AMIT

T-Rex/TESLA

CRS Network Monitoring

PME

at least 5mins 6mins

at most 10mins

We need this kind of technology to describe context


The SmartSwitch PME Beyond traditional CEP

Matching complex patterns beyond CEP We can match complex definitions of situation including

temporal relationships, event correlation and data from

application (CRM, network management, HLR, etc.).

Pattern Query

Language

Algebraic

Transformation

Distributed share-nothing

deployment

(80K ev/s/core)

at least 5mins 6mins

at most 10mins


When to Use Stream

For Fast Analytical & Statistical Jobs The Stream processing layer let you apply a variety of operations

on different event Streams and to compute KPI about users,

services, applications, products or Networks in near RT. The

Stream are processed in real-time according the topologies.

Service ranking

Network failure rate per region

Last week Customer value

AVG Call per region Network monitoring

Real-time BAM

User ranking

Family members or close

friends identification (#Calls

per weeks)

semiocast.com

Example: Knowledge about usage Defining the success of the products by means of patterns

Understanding the usage behaviors

Finding where are the issues

Product Life cycle Evaluation

COUNT &

FILTER


When To Use (Huawei) PME

Help operators detect “interesting

situations” in real-time! We can now implement “context-oriented” software which reacts

according to the current contexts. This is the final achievement of

the full service personalization: the service is not only tailored for

you but it is also triggered when you need it or when it is needed!

Intelligent BPM

Intelligent Process Operations

On-line Marketing campaigns

Marketing automation

Dynamic Network Management

Fraud detection

RT traffic management

Google Now like system

Richer IFTTT services

Intelligent alarm system

Intelligent integration (pattern checking) Event-based campaigns

QoE failure detection

REAL

CONTEXT


But Still, we need a convergent solution What do I need to implement those cases?

Convergence between Batch Processing & Real-time Processing? More exactly between Batch Data Mining & Real-Time Context. OK, but today what are the architectural patterns developed in the internet world that can achieve that ?

Nothing as such exist!! The Nathan Marz’s Lamda architecture is the closest pattern


Let’s come back on the Big Data ambiguity All our use cases are Big Data & Intelligent (Machine Learning)


The Big Data ambiguity But then we lack of technologies to support our Telco Biz. Cases !

The Twitter lambda architecture is located here ! Let’s see how we could extend it


What is the Lambda

Architecture? Description & what is missing for a Telco?


What is the Lambda Architecture? Proposed by the Twitter Storm’s architect

Introduction End of 2012 Nathan Marz published the Lambda architecture description used at Twitter for computing Real-time & Hadoop Batch processing. The idea is that the client can query the batch views (e.g. the number of page views for a web site) and merging these results with the real-time views (e.g. the number of page views during the last 10 mins – duration of the batch processing).

Extending Hadoop Stack with Real-Time views



Architecture view Integration between Query Focused Data Set (QFD) in batch and in Real-time.

But then No way to implement much more complex Processing ? Such as data mining & RT context?



Architecture view Integration between Query Focused Data Set (QFD) in batch and in Real-time.

OK but … This work for counter (SUM, MIN, MAX, etc.) What about data mining & predictive data ?


Defining the Lambda 2.0 Architecture for Telcos To tackle the business I introduced we need much more than counters

Step 1: Introducing the Context awareness Instead of querying only simple aggregated views on large data, the Pattern Matching Engine enables client to query aggregated views, real-time views and contexts! Now applications can use the batch views within specific situations.


Defining the Lambda 2.0 Architecture for Telcos Going further the real-time views proposed by the Twitter’s architect

Step 2: Introducing the Data Mining Layer Instead of querying only simple aggregated views on large data, the In-Memory distributed Machine learning enables to compute predicted model & data analytics and exposing predictive models to client, real-time views and contexts! Now applications can use the batch views within specific situations.


Defining the Lambda 2.0 Architecture for Telcos Example in Real-Time Marketing

Sabri searched for Celine Dion’ s album last week Sabri got 3 M. Jackson’s albums within last month

Sabri searched for 3x M. Carrey’s album last 3 days Sabri is located now in a Music shop

Sabri is going to buy a music album in a shop

These last 2 year Sabri bough 60% of R&B, 15% of Pop and 25% of French Music. The predictive model recommends R&B with a discount between 20 & 25 %

RTD Send an offer by SMS to get R&B music with 20% discount on-line


Defining the Lambda 2.0 Architecture for Telcos Going further the real-time views proposed by the Twitter’s architect

Step 3: Introducing the Feedback loop 1. Feedback on situation after the model

application 2. Incremental Learning to stick to the

reality

Requires a Machine Learning Back-end that can handle this!


High Performance Data Miner SmartMiner

High performance Mining algorithms on 1 Pb Data < 4

mins: is this possible?

High efficiency Increasing prediction accuracy up

to 95%

Incremental learning The predictive model is

continuously processed to fit

the reality

Volume What about applying

algorithms on multiple Pb of

data?


What is missing in current Big Data Frameworks?

Map-Reduce-like models are not

so easy for machine learning

key-based operators limits flexibility

Iterations are complex and computationally

expensive

Only batch learning: Scan all data

Cannot handle Online learning algorithms

No Incremental learning by adding

data piece by piece

• High performance Machine Learning platform

› Combine Online and batch Learning

techniques

• Distributed Machine Learning Architecture

› High scalable and distributed paradigm

› Based on very recent research works

› Architecture-aware algorithms

• Algorithm design based on :

› Building sub-models from disjoint subsets of data.

› Parameter Mixing for computing the final model

• Architecture design :

› in-Memory data storage

› Data locality computation


Huawei Distributed In-Memory Computing

Data Grid

Step 0 : Data

are loaded into

the data grid

Controlle

r

Step 1 :

Launch

learning

signal

Step 1’ : Each Node execute

the training algorithm on data,

locally.

Mixing node

Learner nodes

Step 2 : Each node send,

independently its models

to the mixing node

Step 2’ : The

mixing node

update his model

as the models

arrive

Step 3 :

applying model

to unseen data


Benchmarking

Type Algorithm Parameters

Dataset

Output Size Rows

Column

s

Clustering K-means/BIRCH

① 20 clusters

② Max iter 20

③ precision 1e-6

47GB 40 million 200 Based on same

precision

Classification Logistic Regression

(w/o feature selection)

① w/o feature

selection；

② Max iter 50

③ Precision 1e-6

47GB 40 million 143 Logistic regression

model

Classification Naive Bayes/RFM N/A 30GB 40 million 100 Classification model

Environment

OS Information SUSE Linux Enterprise Server 11 (x86_64)

OS Core Info linux-244 2.6.32.12-0.7-default

Processor Version Intel(R) Xeon(R) CPU E5-2620

Number of Processors 24

Processor Clock Speed 1.6 GHz

CPU Type 64-bit

Memory Size 400 GB

Hard Disk 2 T

Paging file size / Swap space 2GB

Environment: Huawei RH2288 Server：

Test on single node/multi

nodes with single

thread/multi-thread for

performance and speedup

Algorithm SmartMiner 3 SPARK SmartMiner 2 SM3VS SM2

Speedup

K-means 290s 450s 15300s 53x

Naive Bayes 38s 600s 27180s 715x

Log. Regression 400s 75000s 325500s 814x


Think about simulating a marketing Campaign on the entire subscriber base…. In few mins…

Why do you need to be 800x Faster? Is it really needed?


The double bus pattern The Event bus plays the role of the event collector (as defined in EDA), while the Service Bus abstracts the different services that can be used as actions. The workflow engine guarantees that we can have a workflow of actions when a pattern is recognized. The complex event can also be forwarded back on the Event collector for further consumption.

Integration Strategy How the a PME can be deployed and integrated?


Huawei Real-Time Context Blueprint architecture

Event

Detection

Event

Process

H. Perf. Data

Mining

RTD Engine

Data

Sources

Online Channels

SMS

USSD

MMS

IVR

Portals

ODPs

Device

Network

Billing/CRM

External

Channels

EDW

SmartCare

Big Data

Hadoop

Exploration

Analyst CS

PS

Probes

Probes

PME Advanced

CEP

Str

ea

m

Fil

ter

Event

Event

Event

Huawei

Stream

Machine

Learning

Prediction

Modeling

Algorithm Suite

Cust. Knowledge

Mgmt.

Big Data Ingestion

Real-time

Batch ETL

In-

memory

& MPP

Content Analysis

Profiling/Tagging

Segmentation

ETL

Provisioning/

Billing

Real-time

Campaign

Design

Planning

Monitor/Eval

Huawei

Channels

UCM

Toolbar

End User

Telco

Conn

Event

Filter

Decision Engine

Automated

Action

Predefine

d Action

Interaction

Execution

Multi-media

Interaction

Platform

Orchestration

ESB


Examples of Use cases with the PME Illustrations


The Premier league Use case Example: The permission-base marketing – the Premier league Use case

Vs

1. Detect the “Fan” fact (location & context) 2. Detect “A fan is moving far from home to see a match” 3. Detect “A goal has been marked by Manchester”

Permission-based Marketing Soccer Example


The Premier league Use case Example: The permission-base marketing – the Premier league Use case

Vs

1. Detect the “Fan” fact (location & context) 2. Detect “A fan is moving far from home to see a match” 3. Detect “A goal has been marked by Manchester”

Permission-based Marketing Soccer Example

EVENT Goal(Team: string, MatchLocation :string) PARTITION BY (MatchLocation) EVENT UserAbroad(UserID: string, Location: string, Team: string) PARTITION BY (Location) EVENT HotelRoomOffer(UserID:string, Location :string) PARTITION BY (Location) DATA string FavouriteTeam(UserID: string)="" PARTITION BY (UserID) EMIT FreeBeerOffer(UserID:=$id, Location := $loc) SELECT AT LEAST 1 UserAbroad(@UserID = $id, @Location = $loc, GET isFootBallFan(UserID := @UserID) = true) WITHIN 90 MINUTES AS $out AND AT LEAST 1 Goal(@MatchLocation = $loc, @Team = $team) WITHIN LESS THAN 90 MINUTES AFTER $out[0] CHECK $team = GET FavouriteTeam(UserID:=$id)


Defining the Mood of users User Segmentation by their current context and usages

The Use Case Assume that we receive web traffic information and we try to segment users based on that traffic. We assume that there has been prior work to determine the kinds of profile we want to recognize. Say we want to capture “female mood”, defined as “a user who visited female-specific websites three times in the last hour and who searched for female terms two times in the last hour”. Independently of the actual sex of the user, we want to tailor the advertisements we show him to his current mood.

EMIT HappyUser ( UserID := $id ) SELECT AT LEAST 3 Visited ( @UserID = $id, true = GET URLHasTag (URL := @URL, Tag := "Happy" ) ) WITHIN LESS THAN 1 HOUR AS $us AND AT LEAST 2 Searched ( @UserID = $id , true = GET WordHasTag (Word := @Word, Tag := "Happy" ) ) WITHIN LESS THAN 1 HOUR AFTER $us [ 0 ]

Big bang Theory


Product Roll-up Tracking Product usage tracking

Goal One of our Chinese customer wanted to profile a “comic product” in order to define ASAP the best product customization to each consumer. The Business Goal is to customize the product in order to improve customer experience, increase service penetration and increase cross-sales.

A fictional Gaming service In order to show the relevance of the proposition we consider a concrete example of a mobile game resold by the Teclo to Customer. The game is composed by different levels and uses a freemium model. The user can buy artifacts (pets, swords, bike, etc.) and can share it through invitation to friends.

Thanks to “realtime big data”, it is possible to automatically create and apply knowledge about products and consumers in order to maximize the revenue according to the subscriber base behavior and the product catalog.

Real time big data for optimizing service usage & revenue

The Goal is to (1) create and infer knowledge about the games and users, (2) calculate or infer knowledge about game usage, (3) rank the games, (4) Apply this knowledge for a better QoE and increasing sales.


Product Roll-up Tracking Defining the patterns to recognize

Game Success for gamer From the installation date, the user has played at least 10x within 3 days, shared the game with at least 2 friends and bought 2 artifacts.

Game Middle-Success for gamer From the installation date, the user has played between 2 and 5x within 3 days, did not share the and did not bought any artifacts

Game Middle-Success & leaving From the installation date, the user has played between 2 and 5x within 3 days, did not share the and did not bought any artifacts, and did not connect within 2 days after the last play. Or the user has played more than 10x last 2weeks and has not played within 4 days and he is not abroad.

Game Failure From the installation date, the user has played at most 2x within 3 days, did not share the and did not bought any artifacts.

Cannot be expressed by traditional CEP technologies


Business cases that we can now realize

without coding Pattern query example

EMIT Success(GameID := $gId)

SELECT AT LEAST 1 Installed(@GameID = $gId, @UserID = $id) WITHIN 3 DAYS AS $installed

AND AT LEAST 10 Opened(@GameID = $gId, @UserID = $id) WITHIN 3 DAYS AFTER $installed[0]

AND AT LEAST 2 BonusBought(@GameID = $gId, @UserID = $id) WITHIN 3 DAYS AFTER $installed[0]

AND AT LEAST 2 InvitationSent(@GameID = $gId, @UserID = $id) WITHIN 3 DAYS AFTER $installed[0]

EMIT Middle(GameID := $gId)

SELECT AT LEAST 1 Installed(@GameID = $gId, @UserID = $id) WITHIN 3 DAYS AS $installed

AND AT LEAST 10 Opened(@GameID = $gId, @UserID = $id) WITHIN 3 DAYS AFTER $installed[0] as $play

AND NO BonusBought(@GameID = $gId, @UserID = $id) WITHIN 2 DAYS AFTER $play[-1]

AND NO InvitationSent(@GameID = $gId, @UserID = $id) WITHIN 2 DAYS AFTER $play[-1]


Conclusion & Future works (Current R&D) Real-Time Analytics


Conclusion Real-Time Analytics

New Business cases means new Technological Directions The Pattern Matching Engine and an Incremental High performance ML platform can contribute significantly to this paradigm shift.


Future Works & R&D Real-Time Analytics

Automated Learning Data PaaS

Boosting Accuracy with Representation learning/ Deep Learning

Anticipation of situations – Probabilistic Predicted Events (Pattern)

Context learning

Consumer behavior learning

Thank you www.huawei.com

http://twiter.com/sskhiri

https://www.linkedin.com/profile/view?id=6710531

https://github.com/sskhiri


History of CEP over 10Y Starting from Event Stream to data mining

2003

2005 2006

2012

(1) Stream processing only (2) Pattern matching only as Cont. queries

Pattern & state matching

SmartSwitch PME

Data & Analytics

Lambda Architecture 2.0 Convergence between Real-Time Analytics, Context-awareness and Online Learning