38
1 © 2015 The MathWorks, Inc. Get More From Your Data with Data Analytics Francesca Perino

Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

1© 2015 The MathWorks, Inc.

Get More From Your Data

with Data Analytics

Francesca Perino

Page 2: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

2

Page 3: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

3

Page 4: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

4

Page 5: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

5

Buildings have thermodynamic properties

𝜕𝑢

𝜕𝑡− 𝛼

𝜕2𝑢

𝜕𝑥2+𝜕2𝑢

𝜕𝑦2+𝜕2𝑢

𝜕𝑧2= 0

Page 6: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

6

Temperatures change

Page 7: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

7

Electricity demand varies

Page 8: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

8

Humans have comfort bounds

Page 9: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

9

𝜕𝑢

𝜕𝑡− 𝛼

𝜕2𝑢

𝜕𝑥2+𝜕2𝑢

𝜕𝑦2+𝜕2𝑢

𝜕𝑧2= 0

Page 10: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

10

Page 11: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

11

1. Diverse and/or Big Data

2. Advanced Algorithms

3. Deployment

Traits of Data Analytics applications

Page 12: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

12

Why MATLAB?

Develop embedded

systems with analytics

powered functionality

3

Enable Domain

Experts to be

Data Scientists

2

Analytics that increasingly require both business

and engineering data

1

Data Scientist

Embedded Systems

DATA

• Engineering, Scientific, and Field

• Business and Transactional

Business Systems

Develop analytics to run on both enterprise and

embedded platforms

4

Page 13: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

13

Why MATLAB?

Develop embedded

systems with analytics

powered functionality

3

Enable Domain

Experts to be

Data Scientists

2

Data Scientist

Smarter Embedded Systems

DATA

• Engineering, Scientific, and Field

• Business and Transactional

Business Systems

Develop analytics to run on both enterprise and

embedded platforms

4

Analytics that increasingly require both business

and engineering data

1

Page 14: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

14

Business and Engineering Data

“No matter what industry our client is in, and no matter what data they ask us to analyze—text,

audio, images, or video—MATLAB enables us to provide clear results faster.”

Dr. G Subrahmanya VRK Rao, Cognizant

File I/O

• Text

• Spreadsheet

• XML

• CDF/HDF

• Image

• Audio

• Video

• Geospatial

Repositories

• Databases (SQL)

• NoSQL

• Hadoop

File I/O

• Text

• Spreadsheet

• XML

Business and Transactional Data Engineering, Scientific, and Field Data

Web Sources

• HTML

• Mapping

• Financial datafeeds

• RESTful

• JSON

Communication Protocols

• CAN (Controller Area Network)

• DDS (Data Distribution Service)

• OPC (OLE for Process Control)

• XCP (eXplicit Control Protocol)

Real-Time Sources

• Sensors

• GPS

• Instrumentation

• Cameras

• Communication systems

• Machines (embedded systems)

Page 15: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

15

Accessing Data

Page 16: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

16

Datastore

HDFS

Reduce

Node

Node

Node Data

Data

Data

Map

ReduceMap

ReduceMap

Map Reduce

Map

Map

Reduce

Reduce

Page 17: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

17

Table

Page 18: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

18

Preprocessing Data

Page 19: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

19

Signal Processing

cheby2

filter

rms

pwelch

periodogram

xcov

findpeaks

movmean

movstd

Page 20: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

20

Image Processing

Perform image processing, analysis, visualization, and algorithm development

Image display and exploration

Image enhancement

Image analysis

Morphological operations

Image registration

Geometric transformation

ROI-based processing

Page 21: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

21

Feature Engineering – Extracting Information from Data

Data type Common Techniques for Deriving Features

Sensor data Peak analysis

Pulse and transition metrics

Spectral measurements (power, bandwidth, mean frequency,

median frequency)

Image and video data Bag of visual words

HOG (Histogram of Oriented Gradients)

Minimum Eigenvalue algorithm

Local feature descriptors

Edge detection

Transactional data Decomposing timestamps into components (day, month, day of

week, etc.)

Calculation of aggregate values

Page 22: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

22

Why MATLAB?

Develop embedded

systems with analytics

powered functionality

3

Analytics that increasingly require both business

and engineering data

1

Data Scientist

Smarter Embedded Systems

DATA

• Engineering, Scientific, and Field

• Business and Transactional

Business Systems

Develop analytics to run on both enterprise and

embedded platforms

4

Enable Domain

Experts to be

Data Scientists

2

Page 23: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

23

Enabling Domain Experts to be Data Scientists

Symbolic

Computing

Neural

Networks

OptimizationSignal

Processing

Image

Processing

Control

Systems Financial

Modeling

Apps Language

Machine

LearningStatistics

“MATLAB has helped accelerate our R&D and deployment with its robust numerical algorithms, extensive visualization and analytics tools, reliable optimization routines, support for object-oriented programming, and

ability to run in the cloud with our production Java applications.”Borislav Savkovic, BuildingIQ

Page 24: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

24

Apps - Classification Learner app

Page 25: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

25

Redesigned execution engine runs MATLAB code faster

– All MATLAB code can be JIT compiled

– A platform for future improvements

Language - MATLAB Execution Engine

The examples of all FSDA functions have been monitored under R2012a,

R2014b, R2015a, R2015b. Results seem to indicate that, generally, on

computationally intensive mathematical/statistical function R2015b yields

remarkable time improvements.

Page 26: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

26

Why MATLAB?

Enable Domain

Experts to be

Data Scientists

2

Analytics that increasingly require both business

and engineering data

1

Data Scientist

Smarter Embedded Systems

DATA

• Engineering, Scientific, and Field

• Business and Transactional

Business Systems

Develop analytics to run on both enterprise and

embedded platforms

4Develop embedded

systems with analytics

powered functionality

3

Page 27: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

27

Smarter Embedded Systems

INTEGRATION

IMPLEMENTATION

DESIGN

TE

ST

AN

D V

ER

IFIC

AT

ION

RESEARCHREQUIREMENT

S

MCU DSP FPGA ASIC

VHDL,

VerilogC, C++

Environment Models

Physical Components

Algorithms

Structured

Text

PLC

GM

Climate control

Airbus

Battery management

Daimler

Cruise controller

manroland

Printing presses

Sonova

Hearing implantsWeinmann

Transport ventilator

Festo

Industrial robots

FLIR

Thermal imaging

ABB

Smart Grid controller

Page 28: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

28

MATLAB Code Generation

Page 29: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

29

C Code Generation

Page 30: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

30

Why MATLAB?

Develop embedded

systems with analytics

powered functionality

3

Enable Domain

Experts to be

Data Scientists

2

Analytics that increasingly require both business

and engineering data

1

Data Scientist

Smarter Embedded Systems

DATA

• Engineering, Scientific, and Field

• Business and Transactional

Business Systems

Develop analytics to run on both enterprise and

embedded platforms

4

Page 31: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

31

Where does the processing happen?

Devices Business Systems

Visibility Self, Maybe Neighbors Entire System

Power Battery (Low) Grid (Unlimited)

Processing Microprocessor (Low) CPU’s (Unlimited)

Storage Memory (Minimum Buffer) HD’s (Unlimited)

Updates Difficult Easy

Page 32: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

32

Why not transfer all the data?

1. Data privacy concerns

2. Cost of network/storage

3. Power required to transmit data from device (for wireless)

4. Response time

Page 33: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

33

Splitting computation

Page 35: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

35

MATLAB

ExcelAdd-in

Hadoop

MATLAB

Compiler

StandaloneApplication

Deploying Algorithms to Enterprise Systems

MATLAB Compiler enables sharing MATLAB programs without integration programming

MATLAB Compiler SDK provides implementation and platform flexibility for software developers

MATLAB Production Server provides the most efficient development path for secure and

scalable web and enterprise applications

MATLAB

Compiler SDK

C/C++

Java

.NET

MATLABProduction

Server

Python

Page 36: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

36

Enterprise Integration – Forecasting Model

Page 37: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

37

MATLAB Differentiators

Develop embedded

systems with analytics

powered functionality

3

Enable Domain

Experts to be

Data Scientists

2

Analytics that increasingly require both business

and engineering data

1

Data Scientist

Smarter Embedded Systems

DATA

• Engineering, Scientific, and Field

• Business and Transactional

Business Systems

Develop analytics to run on both enterprise and

embedded platforms

4

Page 38: Get More From Your Data with Data Analytics...Experts to be Data Scientists 2 Analytics that increasingly require both business and engineering data 1 ... •Hadoop File I/O •Text

38

Learn More