Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
#PIWorld ©2018 OSIsoft, LLC
DataOps Toolchain for Continuous Control Monitoring
Mahyar SEPEHR
1
#PIWorld ©2018 OSIsoft, LLC
Who is apoIO?
2
We work with our clients to optimize their Digital Journey
• Cloud Architecture • Internet Of Things • Industrial Big Data:
PI System
#PIWorld ©2018 OSIsoft, LLC
A Typical Client and her Use-Cases
3
Laura is a P&U Project Manager and Business Analyst
Water Network
issues
Water leak detection
Energy consumption optimization
Smart Meters Average Consumption
Water Network Quality Control
Behaviour of the consumers, etc.
#PIWorld ©2018 OSIsoft, LLC
Sensors from the Water Network send by GSM values in a non chronological way : Invalidates all rules based on events
Quantity optimisation of substances used on a supply chain : Many timeranges contain exceptions because of Data Quality
4
Business Challenges on Data Quality
#PIWorld ©2018 OSIsoft, LLC
5
Data has imperfections : Welcome to Reality!
03:00 03:10 03:20 03:30 03:40 03:50
Speed
Oil Temp
Oil Temp Speed
Torque 03:00
03:10
03:20
03:30
03:40
03:50
Oil
Tem
p
60
60
60
120
60
Oil
Tem
p
50
50
50
50
45
Spe
ed
10.0
10.2
9.8
10.0
10.1
Spe
ed
9.8
10
10
9.9
9.9
Torq
ue
25
24
25
25
25
Comm Failure
Different start times
Additional Measure
Uneven Spacing
Spike / Out of Range
#PIWorld ©2018 OSIsoft, LLC
Analysis of the problem
6
Lack of direct communication : Business & the Team
More direct and frequent communication between actors
Cycle from conception to execution too long
Organize some short iteration to avoid loosing time on failure
Issues with Data quality discovered far after conception & implementation
Integrate the data quality monitoring
Organisation
Process
Toolchain
Wha
t wen
t wro
ng?
How
can
we
fix it
? ?
?
?
#PIWorld ©2018 OSIsoft, LLC
DevOps Approach : Continuous Improvement
7 © https://en.wikipedia.org/wiki/DevOps_toolchain
#PIWorld ©2018 OSIsoft, LLC
Process : DevOps toolchain
8
Source Code Management
Source Code
Test / Build Continuous Integration
Code Quality Review
Deploy To
Dev. Env.
Deploy To
Staging Env.
Acceptance Testing
Deploy To
Production Env.
Deployment Environment
Configuration Deployment tools
#PIWorld ©2018 OSIsoft, LLC
PI System Project life-cycle: Data Driven Processes
9
10101010101010110011010 0101010101010101010010101010101010101010101010
10101010101010110011010 0101010101010101010010101010101010101010101010 DATA
#PIWorld ©2018 OSIsoft, LLC
Organization Chart
Laura has a new role: Product Owner
10
Security
System Administrator
Data Architect
Product Owner Tech Lead
Ops
Data Engineer
Data Scientist
End-Users
Enterprise Architect
#PIWorld ©2018 OSIsoft, LLC
Agile Management : Rethink the Journey
11
z
DESIG
N
TEST
REVI
EW Hotfix
z z
DESIG
N
TEST
REVIEW
DESIG
N
z
DESIG
N
TEST
REVIEW
TEST
REVIEW
Iteration1 3w
Iteration2 3w
Iteration3 3w
Dem
o Fe
atur
es
Dem
o Fe
atur
es Writes User Stories
Core Features including Data Quality
Launch Launch Launch
#PIWorld ©2018 OSIsoft, LLC
11!
1
2!
3!
4!
5!
16!
Simple Excel DataLink
Design with Asset Analytics
Design an AF SDK based Custom Program
Design a PI WEB API Custom Program
STAR : Self Test And Repair by Code
STAR : Self Test And Repair with PI Integrator
Continuous Control Monitoring : MVP Toolchains
Minimum Viable
Product (MVP)
12
#PIWorld ©2018 OSIsoft, LLC
MVP : STAR with Machine Learning
13
PULL
CLEANSE Data quality
AUGMENT Data aggregation
SHAPE
Model normalization
TRANSMIT Data compatibility
PUSH
PI Integrator for Business Analytics
2 3 4 5 6 Analyse Transform Visualise
Write in to PI System
Report
#PIWorld ©2018 OSIsoft, LLC
14
Risks of DataOps Approach
Count on a clear Data Governance & Support?
Data Exploration managed by Business need
Tasks in iterations depending on other Teams
Does DataOps Team have Testing Background?
#PIWorld ©2018 OSIsoft, LLC
15
CCM bringing Business values From Data Quality to Data Exploration
Data Quality Statistics helping optimize Business rules
Threshold Optimization
Data Exploration helping in Prediction
Data Exploration cleaning useless Monitoring Alerts
#PIWorld ©2018 OSIsoft, LLC
16
Call for Participation! Let’s contribute to a Common Goal: Answer to some sharp questions about PI System Data Exploration
pi-data-exploration
PI WEB API Get Data
Data Preparation
Analysis & Transform
Test
Store into
PI System
Report
1
2
3
4
5
6
#PIWorld ©2018 OSIsoft, LLC
pi-data-exploration Description
Sharp Questions: • Classification Modeling as Leak Detection (no threshhold) • Autofill on Missed Data • Statistics over the data as Missed Data Python Dependencies: • PI-Web-API-Client-Python : PI Client for Python • pandas : Data structure manipulation • numpy : perform calculations over the data
https://github.com/msepehr/pi-data-exploration
17
#PIWorld ©2018 OSIsoft, LLC
Key Takeaways Key Insights Lessons learned from our journey
01Blueprint for success Do not let your transformation initiative fail because of a lack of anticipation on the project due to the data
02Data Exploration, Data Discovery more & more vital in Businesses in any sector
03Agility with a clear Roadmap Iteration on core features is completed with a real vision in a long term
04Empowering End-Users Better and faster communication, interactions with DataOps Team
05New Roles to Integrate PO, Data preparation, Data Scientists, Business Intelligence, Data Visualization
06Automate, automate, automate! From Data Exploration to Business Rules, Industrialize as much as you can
18
#PIWorld ©2018 OSIsoft, LLC
19
Data Exploration makes us grow up…
Let’s grow up together! Mahyar SEPEHR IT Architect [email protected]
#PIWorld ©2018 OSIsoft, LLC
Questions?
Please wait for the microphone
State your name & company
Please rate this session in the mobile app!
Search “OSIsoft” in
your app store
20
#PIWorld ©2018 OSIsoft, LLC
21