View
0
Download
0
Category
Preview:
Citation preview
Dingemans, Bert
2-oktober-2017
Tennet Data PlatformSparx Enterprise Architect for modeling and
design
TDP Architectural Sketches2-oktober-2017
Subjects• Introduction
• General information on the architectural sketches
• Logical TDP Architecture
• Datapipes the elementary building blocks
• Link to generic big data patterns
• Implementation of
• TDP Express
• TDP Advanced
• Reisman use case
Introduction
TDP Architectural Sketches2-oktober-2017
Introduction Bert Dingemans• Independent Data Architect and EA Consultant
• Solution architect for the TenneT Data Platform
• Author/Developer of
• Book Data Architectuur in de praktijk (https://www.bol.com/nl/p/data-
architectuur-in-de-praktijk/9200000027961426)
• Web Publication Platform for Sparx EA (http://wpp.interactory.nl)
• Participant in http://eaxpertise.nl
• Working with EA since version 8 for
• Datamodeling (classes and tables)
• XML message modeling
• ArchiMate 2 & 3 for enterprise architectures and solution
architectures
• Architectural document generation
TDP Architectural Sketches2-oktober-2017
Introduction TenneTTenneT is a leading European electricity transmission system operator
(TSO), with activities in the Netherlands and in Germany. We strive to
ensure a reliable and uninterrupted supply of electricity (99,9999) in
our high-voltage grid for some 41 million people. In doing so, we make
every effort to meet our stakeholders' needs by being responsible,
engaged and connected.
TDP Architectural Sketches2-oktober-2017
Facts & figures• 3040 employees
• 22573 km Grid
• 15 HVDC stations
• 523 Mln Euro profit
• 1848 Mln Euro investments
TDP Architectural Sketches2-oktober-2017
Ou
rg
rid
Dingemans, Bert
2-oktober-2017
DTC and TDP
Data Transformation Corporate
TenneT Data Platform
TDP Architectural Sketches2-oktober-2017
DTC programme
Develop the data space
• Smart meter data
• Production and storage
data (incl LV / MV)
• Grid planning data (incl MV)
• Congestion management
data (incl MV)
• Formally establish vertical leadership
position
• Informally, as a front runner, establish
horizontal leadership
Improve TenneT’s visibility
as a data company that
creates value for society, as
a neutral party
• To support the first work stream,
quickly and visibly take responsibility
for enabling the change by publishing
relevant data in a user friendly manner
• As a second step, create an intelligent
energy market platform: a host for the
new energy eco system
Build our internal data and
analytics capabilities
• Data and IT governance
• Analytics
• Infrastructure
• Run a set of internal business
improvement projects with a heavy
data component, supported by external
specialists, with the aim to enhance
our internal capabilities on data
governance, data management,
analytics, infrastructure
a. Define a new
market design
and develop a direct
business model
b. Lobby for data access
and new market
design
c. Build the
front end of the
intelligent data hub
d. Manage a
portfolio of
projects
e. Scout for best
practices and partners
f. Build TenneT Data
Platform
Team tasksExplanation
TDP Architectural Sketches2-oktober-2017
Traditional data and big data
Data
Ware
House
Traditional
Sources
Hadoop &
Streaming
data
New
Sources
Enterprise
Integration
Structured
Repeatable
Linear
Unstructured
Exploratory
Iterative
Log data
Social data
Situational data
Context data
Sensor data
Public data
Transactional data
ERP data
Internal app data
OLTP data
Traditional (deployed) Big data (discovery)
SCADAMachine data
TDP Architectural Sketches2-oktober-2017
Scope of TDP
Data
Ware
House
Traditional
Sources
Hadoop &
Streaming
data
New
Sources
Enterprise
Integration
Structured
Repeatable
Linear
Unstructured
Exploratory
Iterative
Log data
Social data
Situational data
Context data
Sensor data
Public data
Transactional data
ERP data
Internal app data
OLTP data
Traditional (deployed) Big data (discovery)
SCADAMachine data
Scope of TDP
TDP Architectural Sketches2-oktober-2017
Scope of TDP
Data
Ware
House
Traditional
Sources
Hadoop &
Streaming
data
New
Sources
Enterprise
Integration
Structured
Repeatable
Linear
Unstructured
Exploratory
Iterative
Log data
Social data
Situational data
Context data
Sensor data
Public data
Transactional data
ERP data
Internal app data
OLTP data
Traditional (deployed) Big data (discovery)
SCADAMachine data
Descriptive
analytics
Diagnostic
analytics
Predictive
analytics
Prescriptive
analytics
Scope of TDP
TDP Architectural Sketches2-oktober-2017
TDP development cf. DAMHOF
TDP Architectural Sketches2-oktober-2017
TDP development cf. DAMHOF
DWH
Hadoop
Express
lab
Hadoop
Advanced
lab
MDM
ETL
TDP Architectural Sketches2-oktober-2017
General information on the architectural approach
• For the TDP we have a use case driven approach
• We use an agile approach for developing the platform architecture
and support the use cases
• Just in Time architecting is necessary for the support of use cases
• We create a layered architecture tu support just in time architecting
• We use building blocks for standardisation and modelling speed
• We use ArchiMate as architectural modelling language and UML
class diagrams for information modelling
• We use Sparx Enterprise Architect for continuous architecting
• Architectural documents are generated from the EA repository on
request
TDP Architectural Sketches2-oktober-2017
Use casesTraditional
• MaxLimit
• OptiDat MDM
• Reisman
• Res forecasting
• Ivision (BMW & Volkswagen)
• DLR
Hadoop
Dingemans, Bert
2-oktober-2017
Sparx EA
TDP Architectural Sketches2-oktober-2017
Subjects in EA• Logical architecture
• Big Data Patterns
• ABB
• Hourglass models
• Data pipes
• SBB
• Express
• Advanced
• Use cases
• Reisman
• RES forecasting
• Non functionals
• Dama
• ISO Software qualities
• AIC
TDP Architectural Sketches2-oktober-2017
Layere
darc
hitectu
reO
verv
iew
application Logical integration platform architecture
Data source systems
API & Gateway for data sources
Data integration systems
Data consuming systems
API & Gateway for data integration
Data processing in integration
Data processing in source systems
Data processing in consuming systems
Data storage in integration systems
Data storage in consuming systems
Data production
Data managementData security
Data usage
Data storage in source systems
TDP Architectural Sketches2-oktober-2017
Exam
ple
sto
rag
e application Data storage
Data storage
Relational database
Data warehouse
File system
Distributed fi le system
NoSQL
Key value store Document database Columnar Graph
NewSQLGeographical data
storage
DatamartDatacube
Queues and stacks
Currently relevant
Short term relevant
Low relevance
Legend
TDP Architectural Sketches2-oktober-2017
Exam
ple
inte
rfaces application Data integration interfaces
Data access
Data integration
Application
integration
User interface
Database link
Views and
materialized views
Packages and
stored procedures
Report Dashboard Forms and pages
Webservice
File transfer
Graphs Portals and widgets
Relational interface
B2B integration
UBL etc
ebMS
Programmers API
ODBC
OLEDB and JDBC
etc
ODataEDI
SOAP/XML
webservice
GEO Webservices
TDP Architectural Sketches2-oktober-2017
Exam
ple
man
ag
em
en
t
application Data management
Data management
MDMData quality
management
Data modelingRule mgt
Meta data
management
Data Management
Registry
Data ownership Currently relevant
Short term relevant
Low relevance
Legend
Dingemans, Bert
2-oktober-2017
Link to Big Data Patterns
TDP Architectural Sketches2-oktober-2017
Exam
ple
data
pip
es
application Big Data Pipeline
Big Data Pipeline*
Poly Source* Poly Storage* Big Data Processing
Environment*
Poly Sink* Automated Dataset
Execution
Compound
Required
Optional
Legenda
TDP Architectural Sketches2-oktober-2017
Exam
ple
an
aly
tical
san
db
ox
application Analytical Sandbox
Analytical Sandbox*
Poly Source*
Poly Storage*
Large Scale Batch
Processing
Processing Abstraction
Data Size Reduction
Automated Dataset
Execution
Direct Data Access
Confidential Data
Storage
Large Scale Graph
Processing
Compound
Required
Optional
Legenda
Dingemans, Bert
2-oktober-2017
Data pipes
TDP Architectural Sketches2-oktober-2017
Data
pip
e A
BB
overv
iew
application Serv icemodel ABB
Register extraction
Generic Dataset Access
Use cases en logisch
model uitwerken
wannneer welke
ABB/SBB toepassen
User interface Application
integration
Data integration
Standardized object or
datamodel
Connection requirements
NoSql transformationFile transformation Message
transformation
Relational
transformation
Implementation processes Governance processes
Data Qualities
Streaming
transformation
TDP Architectural Sketches2-oktober-2017
Data
pip
e A
BB
rela
tio
nal application Relation transformation logical architecture ABB
Transformation
Relational transformation
Extract
Transform
Load
Modeling
Logical modeling
Physical modeling
Data target
Data source
Relational databases
Workflow management
TDP Architectural Sketches2-oktober-2017
Data
pip
e S
BB
messag
eapplication XML transformation logical and physical architecture (SBB)
Transformation
Relational transformation
Extract
Transform
Load
Modeling
Logical modeling
Physical modeling
Data target
Data source
Relational databases
Workflow management
XML based
webservice
XML Transformation
XML File
Intermediate XML
storage
Oracle Data
Integrator
Enterprise Edition
Oracle Data
Integrator for Big
Data
Management Pack
for Data integration
Oracle Data
Integrator for
Hadoop
Sparx Enterprise
Architect 12.1
Oracle Service Bus
Dingemans, Bert
2-oktober-2017
TDP Express
TDP Architectural Sketches2-oktober-2017
Lo
gic
alA
rch
itectu
eE
xp
ress application TDP Express logical architecture
Data consuming systems
API & Gateway for data virtualisation
Data processing in consuming systems
Data storage in consuming systems
Data managementData security Data usage
Administration,
Authentication
and authorisation
Anonimisation
and masking
Reporting/BI VisualisationData analytics
File transfer
Data Transformation Event processing Filtering and selection Ordering
Distributed fi le system NoSQL
Data modeling
Meta data
management
TDP Architectural Sketches2-oktober-2017
Ph
ysic
alA
rch
itectu
eE
xp
ress application TDP Express logical and physical architecture
Data consuming systems
API & Gateway for data virtualisation
Data processing in consuming systems
Data storage in consuming systems
Data managementData security
Data usage
Administration,
Authentication
and authorisation
Anonimisation
and masking
Reporting/BI VisualisationData analytics
File transfer
Data Transformation Event processing Filtering and selection Ordering
Distributed fi le system NoSQL
Data modeling
Meta data
management
HUE
Apache Impala
Apache Oozie
Apache Pig
Apache Sentry
Apache Spark
HDFS
Sparx Enterprise
Architect 12.1
R & R Studio
TDP Architectural Sketches2-oktober-2017
Express cloud alternatives
application TDP Express cloud alternativ es
application Hadoop on TPC
Apache Hadoop
Big Data Platform
Service
Hadoop modules
Big Data
infrastructure
Data ProcessingData Storage
Virtualized TPC infrastructure
Storage Device Hadoop Node
configuration
Processing Device
application Hadoop in Public Cloud
Apache Hadoop
Big Data Platform
Service
Public cloud
Big Data
infrastructure
Data ProcessingData Storage
Storage Device Hadoop Node
configuration
Processing Device
Dingemans, Bert
2-oktober-2017
TDP Advanced
TDP Architectural Sketches2-oktober-2017
Lo
gic
alA
rch
itectu
eA
dvan
ced application TDP Adv anced logical architecture
Data consuming systems
API & Gateway for data virtualisation
Data processing in consuming systems
Data storage in consuming systems
Data managementData security Data usage
Administration,
Authentication
and authorisation
Anonimisation
and masking
Reporting/BI
Visualisation
Data analytics
File transfer
Data Transformation Event processing Filtering and selection Ordering
Distributed fi le system NoSQL
Data modeling
Meta data
management
PortalDescriptive
analytics
Predictive analyticsPrescriptive
analytics
Data mining
discovery
Relational databaseGeographical data
storage
Data quality
management
MDM
Data ownership
Data tracing and
lineage
Data protection
(privacy)
WebserviceRelational
interface
Programmers APIGEO Webservices
TDP Architectural Sketches2-oktober-2017
Ph
ysic
alA
rch
itectu
eA
dvan
ced
application TDP Adv anced logical and physical architecture
Data consuming systems
API & Gateway for data virtualisation
Data processing in consuming systems
Data storage in consuming systems
Data managementData security Data usage
Administration,
Authentication
and authorisation
Anonimisation
and masking
Reporting/BI
Visualisation
Data analytics
File transfer
Data Transformation Event processing Filtering and selection Ordering
Distributed fi le system NoSQL
Data modeling
Meta data
management
PortalDescriptive
analytics
Predictive analyticsPrescriptive
analytics
Data mining
discovery
Relational databaseGeographical data
storage
Data quality
management
MDM
Data ownership
Data tracing and
lineage
Data protection
(privacy)
WebserviceRelational
interface
Programmers APIGEO Webservices
HUE
Apache Impala
Apache OozieApache Pig
Apache Sentry
Apache Spark
HDFSOracle Data
Integrator
Enterprise Edition
Oracle Service Bus Streaming analytics
SAP BOR & R Studio
Oracle Golden
Gate
R-Shiny
Sparx Enterprise
Architect 12.1
Dingemans, Bert
2-oktober-2017
Reisman
TDP Architectural Sketches2-oktober-2017
Lo
gic
alA
rch
itectu
re R
eis
man
application Logical Architecture Reisman
Data source systems
API & Gateway for data integration
Data integration systems
Data consuming systems
API & Gateway for data virtualisation
Data processing in integration
Data processing in source systems
Data processing in consuming systems
Data storage in integration systems
Data storage in consuming systems
Data production
Data managementData security
Data usage
File transfer Relational
interface
NLS08
Prophet data
BerichteTool
Markt
stammdaten
VNBSO & PO
VNB Abruf VNB Mess VNB
Abrechnung
Transbill/
Ecount data
EEG
Anlagen
Weather Data
from
Meteorological
Stations
Scada OptiDat
MDM
File transformationRelational
transformationData warehouse
Reporting/BIVisualisation Data analytics Data integration
TDP Architectural Sketches2-oktober-2017
Adaptations and extensions• Web Publication Platform
• Document templates
• Duplication validation and Merge
• Helper routines in DLL
TDP Architectural Sketches2-oktober-2017
Lessons learned• Agile approach is difficult when other teams stay waterfall
• LDA
• IMCB
• Only ArchiMate for modeling is insufficient
• Usage of the quick scan in powerpoint helped a lot
• Architectural maturity levels are different in GE and NL
• Start with a data catalog from the start (in EA?)
• Data Management and Governance are essential for success
• Cooperate with other large programmes (Libra/BMR) within TenneT
TDP Architectural Sketches2-oktober-2017
More information
• http://wpp.interactory.nl is configured with the Big Pattern catalog
• Eap file is available from the zip
http://wpp.interactory.nl/upload/demoen.zip
• Email for more information: bert@interactory.nl
Aansprakelijkheid en auteursrecht TenneT
Disclaimer
Deze powerpoint wordt u aangeboden door TenneT TSO B.V. (“TenneT”). De inhoud ervan - alle teksten,
beelden en geluiden - is beschermd op grond van de auteurswet. Van de inhoud van deze powerpoint
mag niets worden gekopieerd, tenzij daartoe expliciet door TenneT mogelijkheden worden geboden en
aan de inhoud mag niets worden veranderd. TenneT zet zich in voor een juiste en actuele
informatieverstrekking, maar geeft ter zake geen garanties voor juistheid, nauwkeurigheid en
volledigheid.
TenneT aanvaardt geen aansprakelijkheid voor (vermeende) schade, voortvloeiend uit deze powerpoint,
noch voor de gevolgen van activiteiten die worden ondernomen op basis van gegevens en informatie op
deze powerpoint.
TenneT is een toonaangevende Europese netbeheerder (Transmission System Operator,
TSO) met haar belangrijkste activiteiten in Nederland en Duitsland. Met circa 22.000
kilometer aan hoogspanningsverbindingen zorgen we voor een betrouwbare en zekere
elektriciteitsvoorziening aan 41 miljoen eindgebruikers in de markten die we bedienen.
www.tennet.eu
Taking power further
Recommended