Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics...

#PIWorld ©2019 OSIsoft, LLC

1

Using Streaming Views for Real Time Analytics

Rafael Borges

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

3

Today’s Agenda

• Introduction

•What? Why? How?

•OSIsoft and CRISP-DM

•Architecture & Data Flow

•Live Demo

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

4

Before we begin…

•This is a Tech Talk, not a traditional lab or lecture

• It’s informal and interactive

•There will be a Q&A session at the and so we can discuss the content that was presented.

• It will be very dense!

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

5

What, Why and How

•What? • Stream live data to an external server in order to

process it with more powerful analytical tools

•Why? • Because there are platforms that are more capable

for specific tasks and Spark is perfect for HPC

•How? • Using Integrators’ Streaming View, Kafka and Spark

#PIWorld ©2019 OSIsoft, LLC

6 6

OSIsoft and CRISP-DM⠀

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

Cross-industry Standard Process for Data Mining

OSIsoft and CRISP-DM

• Business Understanding

• Data Understanding

• Data Preparation

• Modeling

• Evaluation

• Deployment

7

Source: Kenneth Jensen / Wikipedia

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

OSIsoft and CRISP-DM

8

Data Ingress Products

• Interfaces

• Connectors

• Custom Code

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

OSIsoft and CRISP-DM

9

Data Exploration Products

• PI Vision

• PI Datalink

• Power BI

• Python / R Scripts

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

OSIsoft and CRISP-DM

10

Data Wrangling Products

• PI Datalink

• PI Integrators

• PI Dev Technologies

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

OSIsoft and CRISP-DM

11

Third-Party Products

• Python (SciPy / NumPy)

• R Scripts

• Spark

• Jupyter

• Flink

• Hadoop

• …

#PIWorld ©2019 OSIsoft, LLC

12 12

Architecture & Data Flow⠀

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

13

Architecture & Data Flow

PI Server

Kafka Server

Spark Cluster

Data Generation

PI Integrator for BA

Domain

Node-RED

#PIWorld ©2019 OSIsoft, LLC

14 14

Data Generation⠀

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

15

Data Generation

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

16

Data Generation

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

17

Data Generation

PI Server

Kafka Server

Spark Cluster

Data Generation

PI Integrator for BA

Domain

Node-RED

PI W

eb

AP

I

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

18

Data Generation – PI Web API

def send_data(timestamp, data, variable_info):

body = get_body(timestamp, data, variable_info)

connection = HTTPSConnection(host, context=ssl._create_unverified_context())

connection.request("POST", url, headers=get_header(), body=body)

response = connection.getresponse()

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

19

Data Generation – WebId 2.0 (2017+)

def generate_web_id_by_path(path):

marker = "Ab" #Because I know it's an Attribute

owner_marker = "E" #Because I know it's an Element

if path[0:2] == "\\\\":

path = path[2:]

encoded_path_bytes = path.upper().encode('utf-8')

encoded_string = b64encode(encoded_path_bytes).decode()

encoded_path = encoded_string.strip('=')\

.replace('+', '-').replace('/', '_')

return "P1{}{}{}".format(marker, owner_marker, encoded_path)

#PIWorld ©2019 OSIsoft, LLC

20 20

PI Integrator for BA Advanced Edition⠀

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

21

PI Integrator for BA Advanced Edition

#PIWorld ©2019 OSIsoft, LLC

22 22

Apache Kafka⠀

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

23

Apache Kafka

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

PI Server

Kafka Server

Spark Cluster

Data Generation

PI Integrator for BA

Domain

Node-RED

24

Apache Kafka

Producer

Consumer Consumer

#PIWorld ©2019 OSIsoft, LLC

25 25

Apache Spark⠀

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

26

Apache Spark

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

27

Apache Spark

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

28

Apache Spark

#PIWorld ©2019 OSIsoft, LLC

29 29

Node-RED⠀

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

PI Server

Kafka Server

Spark Cluster

Data Generation

PI Integrator for BA

Domain

Node-RED

30

Node-RED

Producer

Consumer Consumer

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

31

Node-RED

#PIWorld ©2019 OSIsoft, LLC

32 32

Conclusion⠀

#PIWorld ©2019 OSIsoft, LLC

#PIWorld ©2019 OSIsoft, LLC

33

Conclusion

• It’s easy to send data out of the PI System • PI Web API

• PI Integrator for BA • Not only for Kafka, but any platform!

• It’s easy to do real-time analytics with PI data

• It’s easy to bring data back to the PI System

#PIWorld ©2019 OSIsoft, LLC

Speaker Info

34

• Rafael Borges

• Technology Enablement Engineer

• OSIsoft

• rborges@osisoft.com

#PIWorld ©2019 OSIsoft, LLC

Questions?

Please wait for

the microphone

State your

name & company

Please remember

DOWNLOAD THE MOBILE APP

35