TDP Architectural Sketches - Bellekens · Scout for best practices and partners f. Build TenneT Data Platform ... Data Ware House Traditional Sources Hadoop & Streaming data New Sources

Dingemans, Bert

2-oktober-2017

Tennet Data PlatformSparx Enterprise Architect for modeling and

design

TDP Architectural Sketches2-oktober-2017

Subjects• Introduction

• General information on the architectural sketches

• Logical TDP Architecture

• Datapipes the elementary building blocks

• Link to generic big data patterns

• Implementation of

• TDP Express

• TDP Advanced

• Reisman use case

Introduction


Introduction Bert Dingemans• Independent Data Architect and EA Consultant

• Solution architect for the TenneT Data Platform

• Author/Developer of

• Book Data Architectuur in de praktijk (https://www.bol.com/nl/p/data-

architectuur-in-de-praktijk/9200000027961426)

• Web Publication Platform for Sparx EA (http://wpp.interactory.nl)

• Participant in http://eaxpertise.nl

• Working with EA since version 8 for

• Datamodeling (classes and tables)

• XML message modeling

• ArchiMate 2 & 3 for enterprise architectures and solution

architectures

• Architectural document generation

https://www.bol.com/nl/p/data-architectuur-in-de-praktijk/9200000027961426

http://wpp.interactory.nl/

http://eaxpertise.nl/


Introduction TenneTTenneT is a leading European electricity transmission system operator

(TSO), with activities in the Netherlands and in Germany. We strive to

ensure a reliable and uninterrupted supply of electricity (99,9999) in

our high-voltage grid for some 41 million people. In doing so, we make

every effort to meet our stakeholders' needs by being responsible,

engaged and connected.


Facts & figures• 3040 employees

• 22573 km Grid

• 15 HVDC stations

• 523 Mln Euro profit

• 1848 Mln Euro investments


Ou

rg

rid

Dingemans, Bert

2-oktober-2017

DTC and TDP

Data Transformation Corporate

TenneT Data Platform


DTC programme

Develop the data space

• Smart meter data

• Production and storage

data (incl LV / MV)

• Grid planning data (incl MV)

• Congestion management

data (incl MV)

• Formally establish vertical leadership

position

• Informally, as a front runner, establish

horizontal leadership

Improve TenneT’s visibility

as a data company that

creates value for society, as

a neutral party

• To support the first work stream,

quickly and visibly take responsibility

for enabling the change by publishing

relevant data in a user friendly manner

• As a second step, create an intelligent

energy market platform: a host for the

new energy eco system

Build our internal data and

analytics capabilities

• Data and IT governance

• Analytics

• Infrastructure

• Run a set of internal business

improvement projects with a heavy

data component, supported by external

specialists, with the aim to enhance

our internal capabilities on data

governance, data management,

analytics, infrastructure

a. Define a new

market design

and develop a direct

business model

b. Lobby for data access

and new market

design

c. Build the

front end of the

intelligent data hub

d. Manage a

portfolio of

projects

e. Scout for best

practices and partners

f. Build TenneT Data

Platform

Team tasksExplanation


Traditional data and big data

Data

Ware

House

Traditional

Sources

Hadoop &

Streaming

data

New

Sources

Enterprise

Integration

Structured

Repeatable

Linear

Unstructured

Exploratory

Iterative

Log data

Social data

Situational data

Context data

Sensor data

Public data

Transactional data

ERP data

Internal app data

OLTP data

Traditional (deployed) Big data (discovery)

SCADAMachine data


Scope of TDP

Data

Ware

House

Traditional

Sources

Hadoop &

Streaming

data

New

Sources

Enterprise

Integration

Structured

Repeatable

Linear

Unstructured

Exploratory

Iterative

Log data

Social data

Situational data

Context data

Sensor data

Public data

Transactional data

ERP data

Internal app data

OLTP data


SCADAMachine data

Scope of TDP


Scope of TDP

Data

Ware

House

Traditional

Sources

Hadoop &

Streaming

data

New

Sources

Enterprise

Integration

Structured

Repeatable

Linear

Unstructured

Exploratory

Iterative

Log data

Social data

Situational data

Context data

Sensor data

Public data

Transactional data

ERP data

Internal app data

OLTP data


SCADAMachine data

Descriptive

analytics

Diagnostic

analytics

Predictive

analytics

Prescriptive

analytics

Scope of TDP


TDP development cf. DAMHOF


TDP development cf. DAMHOF

DWH

Hadoop

Express

lab

Hadoop

Advanced

lab

MDM

ETL


General information on the architectural approach

• For the TDP we have a use case driven approach

• We use an agile approach for developing the platform architecture

and support the use cases

• Just in Time architecting is necessary for the support of use cases

• We create a layered architecture tu support just in time architecting

• We use building blocks for standardisation and modelling speed

• We use ArchiMate as architectural modelling language and UML

class diagrams for information modelling

• We use Sparx Enterprise Architect for continuous architecting

• Architectural documents are generated from the EA repository on

request


Use casesTraditional

• MaxLimit

• OptiDat MDM

• Reisman

• Res forecasting

• Ivision (BMW & Volkswagen)

• DLR

Hadoop

Dingemans, Bert

2-oktober-2017

Sparx EA


Subjects in EA• Logical architecture

• Big Data Patterns

• ABB

• Hourglass models

• Data pipes

• SBB

• Express

• Advanced

• Use cases

• Reisman

• RES forecasting

• Non functionals

• Dama

• ISO Software qualities

• AIC


Layere

darc

hitectu

reO

verv

iew

application Logical integration platform architecture

Data source systems

API & Gateway for data sources

Data integration systems

Data consuming systems

API & Gateway for data integration

Data processing in integration

Data processing in source systems

Data processing in consuming systems

Data storage in integration systems

Data storage in consuming systems

Data production

Data managementData security

Data usage

Data storage in source systems


Exam

ple

sto

rag

e application Data storage

Data storage

Relational database

Data warehouse

File system

Distributed fi le system

NoSQL

Key value store Document database Columnar Graph

NewSQLGeographical data

storage

DatamartDatacube

Queues and stacks

Currently relevant

Short term relevant

Low relevance

Legend


Exam

ple

inte

rfaces application Data integration interfaces

Data access

Data integration

Application

integration

User interface

Database link

Views and

materialized views

Packages and

stored procedures

Report Dashboard Forms and pages

Webservice

File transfer

Graphs Portals and widgets

Relational interface

B2B integration

UBL etc

ebMS

Programmers API

ODBC

OLEDB and JDBC

etc

ODataEDI

SOAP/XML

webservice

GEO Webservices


Exam

ple

man

ag

em

en

t

application Data management

Data management

MDMData quality

management

Data modelingRule mgt

Meta data

management

Data Management

Registry

Data ownership Currently relevant

Short term relevant

Low relevance

Legend

Dingemans, Bert

2-oktober-2017

Link to Big Data Patterns


Exam

ple

data

pip

es

application Big Data Pipeline

Big Data Pipeline*

Poly Source* Poly Storage* Big Data Processing

Environment*

Poly Sink* Automated Dataset

Execution

Compound

Required

Optional

Legenda


Exam

ple

an

aly

tical

san

db

ox

application Analytical Sandbox

Analytical Sandbox*

Poly Source*

Poly Storage*

Large Scale Batch

Processing

Processing Abstraction

Data Size Reduction

Automated Dataset

Execution

Direct Data Access

Confidential Data

Storage

Large Scale Graph

Processing

Compound

Required

Optional

Legenda

Dingemans, Bert

2-oktober-2017

Data pipes


Data

pip

e A

BB

overv

iew

application Serv icemodel ABB

Register extraction

Generic Dataset Access

Use cases en logisch

model uitwerken

wannneer welke

ABB/SBB toepassen

User interface Application

integration

Data integration

Standardized object or

datamodel

Connection requirements

NoSql transformationFile transformation Message

transformation

Relational

transformation

Implementation processes Governance processes

Data Qualities

Streaming

transformation


Data

pip

e A

BB

rela

tio

nal application Relation transformation logical architecture ABB

Transformation

Relational transformation

Extract

Transform

Load

Modeling

Logical modeling

Physical modeling

Data target

Data source

Relational databases

Workflow management


Data

pip

e S

BB

messag

eapplication XML transformation logical and physical architecture (SBB)

Transformation

Relational transformation

Extract

Transform

Load

Modeling

Logical modeling

Physical modeling

Data target

Data source

Relational databases

Workflow management

XML based

webservice

XML Transformation

XML File

Intermediate XML

storage

Oracle Data

Integrator

Enterprise Edition

Oracle Data

Integrator for Big

Data

Management Pack

for Data integration

Oracle Data

Integrator for

Hadoop

Sparx Enterprise

Architect 12.1

Oracle Service Bus

Dingemans, Bert

2-oktober-2017

TDP Express


Lo

gic

alA

rch

itectu

eE

xp

ress application TDP Express logical architecture


API & Gateway for data virtualisation



Data managementData security Data usage

Administration,

Authentication

and authorisation

Anonimisation

and masking

Reporting/BI VisualisationData analytics

File transfer

Data Transformation Event processing Filtering and selection Ordering

Distributed fi le system NoSQL

Data modeling

Meta data

management


Ph

ysic

alA

rch

itectu

eE

xp

ress application TDP Express logical and physical architecture






Data usage

Administration,

Authentication

and authorisation

Anonimisation

and masking

Reporting/BI VisualisationData analytics

File transfer



Data modeling

Meta data

management

HUE

Apache Impala

Apache Oozie

Apache Pig

Apache Sentry

Apache Spark

HDFS

Sparx Enterprise

Architect 12.1

R & R Studio


Express cloud alternatives

application TDP Express cloud alternativ es

application Hadoop on TPC

Apache Hadoop

Big Data Platform

Service

Hadoop modules

Big Data

infrastructure

Data ProcessingData Storage

Virtualized TPC infrastructure

Storage Device Hadoop Node

configuration

Processing Device

application Hadoop in Public Cloud

Apache Hadoop

Big Data Platform

Service

Public cloud

Big Data

infrastructure

Data ProcessingData Storage

Storage Device Hadoop Node

configuration

Processing Device

Dingemans, Bert

2-oktober-2017

TDP Advanced


Lo

gic

alA

rch

itectu

eA

dvan

ced application TDP Adv anced logical architecture






Administration,

Authentication

and authorisation

Anonimisation

and masking

Reporting/BI

Visualisation

Data analytics

File transfer



Data modeling

Meta data

management

PortalDescriptive

analytics

Predictive analyticsPrescriptive

analytics

Data mining

discovery

Relational databaseGeographical data

storage

Data quality

management

MDM

Data ownership

Data tracing and

lineage

Data protection

(privacy)

WebserviceRelational

interface

Programmers APIGEO Webservices


Ph

ysic

alA

rch

itectu

eA

dvan

ced

application TDP Adv anced logical and physical architecture






Administration,

Authentication

and authorisation

Anonimisation

and masking

Reporting/BI

Visualisation

Data analytics

File transfer



Data modeling

Meta data

management

PortalDescriptive

analytics

Predictive analyticsPrescriptive

analytics

Data mining

discovery

Relational databaseGeographical data

storage

Data quality

management

MDM

Data ownership

Data tracing and

lineage

Data protection

(privacy)

WebserviceRelational

interface

Programmers APIGEO Webservices

HUE

Apache Impala

Apache OozieApache Pig

Apache Sentry

Apache Spark

HDFSOracle Data

Integrator

Enterprise Edition

Oracle Service Bus Streaming analytics

SAP BOR & R Studio

Oracle Golden

Gate

R-Shiny

Sparx Enterprise

Architect 12.1

Dingemans, Bert

2-oktober-2017

Reisman


Lo

gic

alA

rch

itectu

re R

eis

man

application Logical Architecture Reisman

Data source systems

API & Gateway for data integration

Data integration systems



Data processing in integration

Data processing in source systems


Data storage in integration systems


Data production


Data usage

File transfer Relational

interface

NLS08

Prophet data

BerichteTool

Markt

stammdaten

VNBSO & PO

VNB Abruf VNB Mess VNB

Abrechnung

Transbill/

Ecount data

EEG

Anlagen

Weather Data

from

Meteorological

Stations

Scada OptiDat

MDM

File transformationRelational

transformationData warehouse

Reporting/BIVisualisation Data analytics Data integration


Adaptations and extensions• Web Publication Platform

• Document templates

• Duplication validation and Merge

• Helper routines in DLL


Lessons learned• Agile approach is difficult when other teams stay waterfall

• LDA

• IMCB

• Only ArchiMate for modeling is insufficient

• Usage of the quick scan in powerpoint helped a lot

• Architectural maturity levels are different in GE and NL

• Start with a data catalog from the start (in EA?)

• Data Management and Governance are essential for success

• Cooperate with other large programmes (Libra/BMR) within TenneT


More information

• http://wpp.interactory.nl is configured with the Big Pattern catalog

• Eap file is available from the zip

http://wpp.interactory.nl/upload/demoen.zip

• Email for more information: [email protected]

http://wpp.interactory.nl/

http://wpp.interactory.nl/upload/demoen.zip

mailto:[email protected]

Aansprakelijkheid en auteursrecht TenneT

Disclaimer

Deze powerpoint wordt u aangeboden door TenneT TSO B.V. (“TenneT”). De inhoud ervan - alle teksten,

beelden en geluiden - is beschermd op grond van de auteurswet. Van de inhoud van deze powerpoint

mag niets worden gekopieerd, tenzij daartoe expliciet door TenneT mogelijkheden worden geboden en

aan de inhoud mag niets worden veranderd. TenneT zet zich in voor een juiste en actuele

informatieverstrekking, maar geeft ter zake geen garanties voor juistheid, nauwkeurigheid en

volledigheid.

TenneT aanvaardt geen aansprakelijkheid voor (vermeende) schade, voortvloeiend uit deze powerpoint,

noch voor de gevolgen van activiteiten die worden ondernomen op basis van gegevens en informatie op

deze powerpoint.

TenneT is een toonaangevende Europese netbeheerder (Transmission System Operator,

TSO) met haar belangrijkste activiteiten in Nederland en Duitsland. Met circa 22.000

kilometer aan hoogspanningsverbindingen zorgen we voor een betrouwbare en zekere

elektriciteitsvoorziening aan 41 miljoen eindgebruikers in de markten die we bedienen.

www.tennet.eu

Taking power further

Documents

TDP Architectural Sketches - Bellekens · Scout for best practices and partners f. Build TenneT Data Platform ... Data Ware House Traditional Sources Hadoop & Streaming data New Sources