Embed Size (px)
Citation preview
Beyond Stream Analytics
Ricardo [email protected]
Joining different technologies and smart people together to solve real problems
Based on a real case application on a major operator
Intelie is a data technology company originally founded in Brazil, with presence in Rio de Janeiro, São Paulo, Macaé and Houston.
Our mission is to optimize critical operations through technological solutions specialized in stream data analytics associated with high data volumes and high data throughput.
Since the beginning, Intelie has built a solid customer base by delivering results in high-profile clients in different industries and countries like USA, Malaysia, China and Brazil.
INTELIE
STREAM ANALYTICS INTO PRACTICE
Libra oil field is a large ultra-deepwater oil prospect located in the Santos Basin, about 230 kilometres (140 mi) off the coast of Rio de Janeiro, Brazil. The most probable estimate being 7.9 billion barrels.
Petrobras (operator, with 40%), Shell (20%), Total (20%), CNOOC (10%) and CNPC (10%)
Real-time monitoring and analytics for well operations (drilling, cementing, interventions, completion)
Case started at R&D center and are being used for Libra field exploration
LEARNING FROM THE TRENCHES
Empower engineers and R&D:• Flexibility to create new analysis and visualization without the
need of software development• Extensibility to reduce the lead time of new data-related
heuristics, algorithms, visualization and applications;
Make the best tools and people to collaborate:• There is no silver bullet. The best tools to address the each real
problem in corporation scale must work together and collaborate.
• Make it easier for people to consume and collaborate.
Join engineering approach with machine learning approach:• Use what is best for each problem;• Try new approach to problems;• When possible combine them into one solution
Stream analytics definition:
Software that can filter, aggregate, enrich, and analyze a high throughput of data from multiple, disparate live data sources and in any data format to identify simple and complex patterns to provide applications with context to detect opportune situations, automate immediate actions, and dynamically adapt.
STREAM ANALYTICS - REAL TIME BIG DATA
The Forrester Wave™: Big Data Streaming Analytics, Q1 2016
STREAM ANALYTICS - the upside-down database
The upside-down database analogy
...
data
query 1 query n
continuousquery n
continuousquery 1 ...
relational model stream model
datadata
STREAM ANALYTICS - Deeper understanding
Babcock, Brian, et al. "Models and issues in data stream systems."Proceedings of the twenty-first ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems. ACM, 2002.
ARCHITECTURAL VIEW
STREAM ANALYTICS FOR DRILLING
pressure woh
flow
continuousqueries
rop
temp
gama-raybit depth
WITSMLprotocol Data preparation
Data distribution
Stream intelligence
Simple analysis
Patterns
Machine Learning
Visualization
Read data from rigs and wells using industry protocol (WITSML)
Ensure the right mnemonics semantic and units use
Manage and notify data suppliers promptly
DATA CAPTURE, NORMALIZATION AND QUALITY
Data preparation
Service companies have different names for variables and different units;
Hundreds of sensors per rig, coming from different systems;
Communication stalls, sensors breaks, service fails, ...
Challenges faced
Mapping and identifying changes
Event-driven and pub / sub
Data fusion, quality monitoring
Solution
IN-MEMORY DATA STREAM ANALYSIS ENGINE
Enabling user defined patterns
INTELIE PIPESA specific language and in-memory engine to analyze data stream from sensors.
Filter rig42
Detect pressure increase (A)
Detect Bit Depth decrease (B)
(A) AND (B)
Chained logic model
Stream intelligence
IN-MEMORY DATA STREAM ANALYSIS ENGINE
Enabling user defined patterns
INTELIE PIPESA specific language and in-memory engine to analyze data stream from sensors.
rig42
=> [ @filter \mnemonic:SPP
=> avg(value#) as last_pressure every 20 minutes
=> expand *, _ > :prev(1) and :prev(1) > :prev(2) and :prev(2) > :prev(3) aspressure_increase,
:prev(3) as first_pressure, last_pressure-:prev(3) as pressure_increase_amount
=> expand last(*) every 5 minutes
join
@filter \mnemonic:DBTM
=> avg(value#) as last_bit_depth every 5 minutes
=> expand *, :prev - _ >= 5 and as bit_depth_decrease, :prev as first_bit_depth, :prev- _ as bit_depth_decrease_amount]
=> @filter bit_depth_decrease and pressure_increase
filter rig42
bit depth decrease (B)
pressure increase (A)
A AND B
Stream intelligence
DISTRIBUTED QUERY FOR REAL-TIME ADVANCED ANALYTICS
Sensor
DISTRIBUTED QUERY FOR REAL-TIME ADVANCED ANALYTICS
MAP
REDUCERREDUCERREDUCER
HyperLogLog1 probabilistic data structure to optimize memory consumption
1- Flajolet, P.; Fusy, E.; Gandouet, O.; Meunier, F. (2007). "HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm“. AOFA ’07: Proceedings of the 2007 International Conference on the Analysis of Algorithms.
Hidden complexity
VISUAL EXTENSIBILITY
Enabling user defined visualizationStream intelligence
The “look what I did” effect!
ANALITICAL EXTENSIBILITY
Weight On Bit
Hole Depth
Rotary Speed
Rate Of Penetration
Heave Bit Depth
Flow Rate
Block Position
Weight On Hook
Custom classification
algorithm, extending
Intelie PIPES
INPUT SIGNALS IDENTIFIED OPERATION
Enabling user defined patterns and visual extensibility
Using Java write the algorithm to extend Intelie PIPE functions
Stream intelligence
Let Phd´s and data scientist focus on the core
ANALITICAL EXTENSIBILITY
Enabling user defined patterns and visual extensibility Stream intelligence
MACHINE LEARNING INTO PRACTICE
Mud weight prediction for pre-salt drilling zones
Stream analytics can prepare data and embed powerful algorithms
Data Stream Anomaly Detection through Principal Subspace Tracking
*independent software
Stream intelligence
*not applied in O&G yet
STREAM ANALYTICS + MACHINE LEARNING
Stream intelligence
“The main area of research for the future is to investigate the possibility to build a learning module to detect anomalies in an unsupervised manner, as proven by the HOLMES project”
Kazarov, A., G. Lehmann Miotto, and L. Magnoni. "The AAL project: automated monitoring and
intelligent analysis for the ATLAS data taking infrastructure." Journal of Physics: Conference
Series. Vol. 368. No. 1. IOP Publishing, 2012.
CUSTOM APP EXAMPLE
BOP monitoring and right EDS selection control
loading ...
Stream intelligence
BOPs Become the Focus of Data-Driven Scrutiny
(Data-Driven BOP on July JPT edition)
APPs INTEGRATION
Pronova integrationData distribution
(APPs INTEGRATION)
Slack application example on Github
Data distribution
https://github.com/intelie/plugin-slack
APPs INTEGRATION
PWDa (Petrobras in-house anomaly detection software) two-way integration
Data distribution
APPs INTEGRATION
Data distribution
Intelie Live Platform
Python Application
Matlab set ofalgorithms
ProNova
Pub/Sub
WE
B M
AN
AG
EM
EN
T
streaming data
Real-time analytics in O&G / E&P
“Simple” foundation problems like data integration, data quality, data governance and monitoring still have to be solved. Do not underestimate them.
Empower, using a platform approach, your company, or suppliers, engineers, devs and data scientist to solve real-time analytics related problems. It´s all about good people doing their jobs.
Embrace the best tools in the market to solve specific analytics related problems. Make them also to collaborate,
There is no artificial intelligence silver bullet.
What I have learned... and would like for you to keep in mind!
Future
Drone Pilot
Jet Pilot
