How to Apply Big Data Analytics and Machine Learning to Real Time Processing - Kai Waehner -...

Howtoapplymachinelearningtoreal-timeprocessing

Kai Waehner

MILAN 25-26 NOVEMBER 2016

TechnologyEvangelistkontakt@kai-waehner.deLinkedIn@KaiWaehnerwww.kai-waehner.de

Apply Big Data Analytics to Real Time Processing

Analyze and Act on Critical Business Moments

Agenda

1) Machine Learning and Big Data Analytics

2) Building an Analytic Model

3) Real Time Processing

4) Live Demo

Agenda

4) Live Demo

Machine Learning

…. allows computers to find hidden insights without being explicitly programmed where to look.

Real World Examples of Machine Learning

Spam Detection Search Results +Product Recommendation

Picture Detection(Friends, Locations, Products)

Machine Learning is already present in daily life…

Now, every enterprise is beginning to leverage it!

The Next Disruption:Google Beats Go Champion

Analytics Maturity Model

Immediate Long-TermCompetitiveAdvantageValue to the Organization

A good Big Data Analytics platform can provide value to the organization across the full spectrum of use cases

Self-serviceDashboards

EventProcessingAdvancedAnalytics

Measure Diagnose Predict Optimize Alert Automate

Analytics Maturity

VisualAnalytics

EventProcessing

Analytics

VisualAnalytics

Analytics Maturity

Analytics

Analytics Maturity

VisualAnalytics

EventProcessing

Analytics

Agenda

4) Live Demo

Analytical Pipeline

Analytics Maturity

VisualAnalytics

EventProcessing

Analytics

Analytical Pipeline

Variety of Data in Enterprises

CustomGUI-drivendataaccessvia

SiebeleBusiness

Localdatasources

AccessExcel STDF

Drag-and-drop

SQLServerOracle

InformationServices(join,transform,reusable,

parameterized,dynamicqueryforin-memoryuse)

Databases

JDBC/ODBC

HadoopSFDC

PostgreSQL

TeradataNetezza

Etc.XML

FlatFiles

Spread-sheets

WebServices

OracleE-Business

RDBMSRDBMS

SAP BWSAP R/3 DATA

FABRIC

Salesforce

ODBCOLEDBSqlClient

Directconnection

Oracle

TeradataAsterMSSSASTeradata

DirectQuery(dynamicallyqueryandretrievedata

forvisualizationandanalysis)

Databases

MySQLEtc.

NetezzaHadoop

Data Acquisition

“Smart Recommendation Engine”

Analytical Pipeline

cust_id dept sku dollar gift date1 104 C 12003 2.40 FALSE 2016-10-172 105 A 12005 62.85 FALSE 2016-10-173 102 C 12007 69.23 TRUE 2016-10-174 104 B 12004 9.33 FALSE 2016-10-185 105 C 12010 14.16 TRUE 2016-10-186 101 B 12003 90.43 FALSE 2016-10-197 103 C 12005 90.97 FALSE 2016-10-19n … … … … … …

cust_id A B C total # orders first_date

last_date

1 100 21.76 23.67 0.00 45.43 2 2016-10-19

2016-10-20

2 101 0.01 74.65 0.00 74.66 3 2016-10-19

2016-10-20

3 102 0.00 60.92 50.29 111.21 6 2016-10-17

2016-10-20

4 103 0.00 0.00 52.30 52.30 2 2016-10-19

2016-10-20

5 104 31.34 9.33 2.40 43.06 4 2016-10-17

2016-10-20

6 105 62.85 0.00 56.00 118.85 3 2016-10-17

2016-10-20

Data Munging - Transformations

Analytical Pipeline

“The greatest value of a picture is when it forces us to notice what we never expected to see”

John W. Tukey, 1977

Exploratory Data Analysis

Visual Analytics - Interactive Brush-Linked

… and “Inline Data Wrangling” à Ad-hoc data preparation instead of just ETL

VisualAnalytics

Analytics Maturity

Analytics

Analytical Pipeline

Which picture represents a model?

A model is a simplification of the truth that helps you with decision making.

Model Building

Employees who write longer emails earn higher salaries!

Model Building

Model Improvement

Managers

Model Improvement

Analytical Pipeline

Model Validation

How is the IQ of a kid related to the IQ of his / her mum?

Frameworks and Tooling

Advanced Analytics and Big Data Tools (for Data Scientists)

Many more ….

“…as a next-generation data discovery capability that automatically finds and explains insights from advanced analytics to business users or citizen data scientists”

Smart Data Discovery (for the Business User)

Leverage Machine Learningwithout the help of a Data Scientist

Smart Visual Analytics vs. Data Science Tools

Live DemoLive Demo

Agenda

4) Live Demo

Analytics Maturity

VisualAnalytics

EventProcessing

Analytics

Traditional Data Processing: ”Request – Response”

Analyze

The New Era: Streaming Analytics

Act & Monitor

Analyze

Streaming Analytics - Processing Pipeline

Adapters / Channels

Integration

Messaging

Stream Ingest

Transformation

Aggregation

Enrichment

Filtering

StreamPreprocessing

Process Management

Analytics (Real Time)

Applications& APIs

Analytics / DW

Reporting

StreamOutcomes

• Contextual Rules

• Windowing

• Patterns

• Analytics

• Deep ML

• …

Stream Analytics & Processing

Index / SearchNormalization

Applying an Analytic Modelis just a piece of the puzzle!

Frameworks and Products

(no complete list!)

OPEN SOURCE CLOSED SOURCE

PRODUCT

FRAMEWORK

Azure MicrosoftStream Analytics

Comparison of Stream Processing Frameworks and Products

Slide Deck and Video Recording:http://www.kai-waehner.de/blog/2016/11/15/streaming-analytics-comparison-open-source-frameworks-products-cloud-services/

Apache Storm – Hello World

http://wpcertification.blogspot.ch/2014/02/helloworld-apache-storm-word-counter.html

Visual Coding for Streaming Analytics

• StreamingOperators• Connectivity• VisualDevelopment• Testing&Simulation• MatureTooling/Support• MiddlewareIntegration

Live Visual Analytics UI

Dynamicaggregation

Livevisualization

Ad-hoccontinuousquery

Alerts

Action

How to apply analytic models to real time processing without redevelopment?

StreamProcessi

ngH20.ai

Open Source R

TERRSpark

ML MATLAB

TIBCO StreamBase Connector for H2O.ai

Agenda

4) Live Demo

Scenario: Predictive Scrapping of Parts in an Assembly Line

Goal: Scrap parts as early as possible automatically to reduce costs in a manufacturing process.

Question: When to scrap a part in Station 1 instead of doing re-work or sending it to Station 2?

Station 1 Station 2

Cost Before9€ 7€ 13€ Total Cost

29€(or more)

Scrap? Scrap?

Fast Data Architecture for Predictive Maintenance

OperationalAnalytics

OperationsLiveUI

CSV Batch

JSON Real Time

XML Real Time

StreamingAnalyticsAction

Aggregate

Analytics

Correlate

LiveDatamart

Continuousqueryprocessing

Alerts

Manualaction,escalation

HISTORICALANALYSIS DataScientists

FlumeHDFS

Spotfire

R/TERRHDFS

Hadoop (Cloudera)

StreamBase

TIBCO Fast Data Platform

OracleRDBMS

Avro Parquet … PMML

InternalData

TIBCO Spotfire with H2O Integration

Data Discovery / Data Mining (“Are parts that repeat a station more likely scrap parts?”)

TIBCO Live Datamart

Operational Intelligence (“Monitor the manufacturing process and change rules in real time!”)

Live Dartmart Desktop Client

TIBCO Live Datamart

Operational Intelligence (“Monitor the manufacturing process and change rules in real time!”)

Live Dartmart Web API

TIBCO Spotfire + StreamBase + H2O.ai + Live Datamart

Live DemoLive Demo

TIBCO Accelerator for Apache Spark

1. Fast Data Preparation for IoTDozens of enterprise and IoT data preparation adapters: MQTT, Databases; inbound creation of HDFS, Parquet, Hbase, Avro…

2. Spotfire Model Discovery TemplateUse Spotfire to explore Spark data lake, create predictive model, train in H20, and deploy to Streaming Analytics.

3. Operationalize Predictive ModelsZookeeper deployment to StreamBase nodes living in Spark cluster via H20, PMML, TERR models

4. Streaming Analytics for AutomationAutomate action based on predictive models – make offers to customers, stop fraudulent transactions, alert.

5. Monitor & Retrain Model Monitor behavior of model, retrain when necessary.

6. Drag & Drop for Business Solution DevelopersCode-free development environment for work with H20, HDFS, Avro, TERR

The TIBCO Accelerator for Spark is a TIBCO engineered, light-weight open-source fast-start for systems to stream data into Spark, discover patterns in Spark with Spotfire, and operationalize the insights on Big Data.

FUNCTIONAL COMPONENTS

Key Take-Aways

Ø Insights are hidden in Historical Data on Big Data Platforms

Ø Machine Learning and Big Data Analytics find these Insights by building Analytics Models

Ø Event Processing uses these Models (without Redevelopment) to take Action in Real Time

Questions? Please contact me!

Kai WähnerTechnology Evangelist at TIBCO

kontakt@kai-waehner.de@KaiWaehnerwww.kai-waehner.deLinkedIn

How to Apply Big Data Analytics and Machine Learning to Real Time Processing - Kai Waehner -...

Technology

(Codemotion 2014) JVM GC: WTF?!

Apache DeviceMap - codemotion Rome 2015

JAZOON'13 - Kai Waehner - Hadoop Integration

Codemotion twitter programming

Ecs codemotion

Codemotion Milan 2015 Alerts Overload

David pilato codemotion-telaviv

Codemotion 2015 Infinispan Tech lab

Codemotion Rome 2015. GlusterFS

Codemotion fuse presentation

Codemotion 2015

OrientDB Codemotion 2014

Groovy programadores Java - Codemotion 2012

Think horizontally @Codemotion

Codemotion 2015 - Unit Testing

Codemotion how to create a product copy - Codemotion Rome 2015

Codemotion provinciali v. 0.1- Codemotion Rome 2015

Unleashing Apache Kafka and TensorFlow in Hybrid Architectures … · 2020. 7. 9. · Apache Kafka and Machine Learning –Kai Waehner 2 Poll In which of these environments, if at

Arquitecturas de microservicios - Codemotion 2014

Codemotion 2013 scalatra_play_spray