Future-Proofing Your Data Strategy - SimplicityBI · Who is using it has changed: everyone How they’re using it has changed: continuously When they’re using it has changed: instantaneously

Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands. All rights reserved. No part of this material may be reproduced, stored

in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photographic, or otherwise, without the explicit

written permission of the copyright owners.

Future-Proofing Your Data Strategy

Rick F. van der LansIndustry analyst

Email [email protected] Twitter @rick_vanderlanswww.r20.nl

Copyright © 2017 R20/Consultancy B.V., The Hague, The Netherlands 2

The Data Center in the Cellar


Data for the Happy Few Only


The First Data Coming Out of the Cellar


So Many New Technologies, Ideas, Architectures, Techniques, and Opportunities


Data

Management

Technology

Usa

ge

Ch

arac

teri

stic

s

Data governance Data protection

Master data

Analytics

Data lake

Data science

Self-Service

Customer

Facing app

Data brokerage

Micro services

Artificial Intelligence

Robotica

Big data

Fast data

Dark data

IoT

Streaming analytics

+AI

+AI

Cloud computing

VR + AR

Block chaining

Hadoop NoSQL

MPP



Don’t Always Believe What You Hear


Don’t Always Believe What You See


Don’t Always Believe What You Read

Big data: A revolution that will transform how we live, work and think

Companies are being destroyed and created around big data, …

Without big data, you are blind and deaf in the middle of a freeway.

Big data has arrived and is shaping IT today

The disruptive power of big data


Data hasn’t changed,

it’s just more of the same


Data usage has changed:Who is using it has changed: everyone

How they’re using it has changed: continuouslyWhen they’re using it has changed: instantaneously

Complexity of usage has changed: analyticsWhich data they need has changed: all


ETL ETLETL

Sourcesystems

Data martsStagingarea

Analytics &reporting

Datawarehouse

The Classic Data Warehouse Architecture


Data Warehouse and Data Usage

Developers

IT specialists

Business Users

Development Styles

Pre-programmed, auditable, governable,

formally tested

Self-service, investigative

Pre-programmed


Report Types

Batch and online business reports

Customer-facing apps

Ad-hoc reports

Simple data retrieval

Ad-hoc reports

Data mining, statistics

Dark data analysis

Consumers

Business users

Legislators

External parties

Consumers

Business users

Business users

Business users

Data scientists

Business users and IT


Phase 1: Data Warehouse and Data Usage

Developers

IT specialists

Business Users

Development Styles


formally tested


Pre-programmed


Report Types



Ad-hoc reports


Ad-hoc reports


Dark data analysis

Consumers

Business users

Legislators

External parties

Consumers

Business users

Business users

Business users

Data scientists




Developers

IT specialists

Business Users

Development Styles


formally tested


Pre-programmed


Report Types



Ad-hoc reports


Ad-hoc reports


Dark data analysis

Consumers

Business users

Legislators

External parties

Consumers

Business users

Business users

Business users

Data scientists




Developers

IT specialists

Business Users

Development Styles


formally tested


Pre-programmed


Report Types



Ad-hoc reports


Ad-hoc reports


Dark data analysis

Consumers

Business users

Legislators

External parties

Consumers

Business users

Business users

Business users

Data scientists




Object Detection in Images


Automatic Image Caption Generation


Retail ExampleFully integrated face detection, real-time content targeting, and statistics gatheringFace tracking and “just-in-time” content deliveryGender recognitionAge estimationEmotion detection (A/B testing and feedback gathering)Clothes size and color detection


Our Users are Changing


ETL ETLETL

Sourcesystems

Data martsStagingarea


Datawarehouse

The Classic Data Warehouse Architecture


Programming logic to verify, transform, cleanse, integrate, interpret, and standardize source data from production systems

Backup and recovery mechanisms

Security mechanisms and policies to protect against unauthorized access and misuse of the data and reports

A priority scheme for developing and maintaining reports

Manual procedures initiated by calamities

The monitoring of reporting performance, scalability, availability and other non-functional aspects of the operational reporting environment

Various administrative procedures related to human and computer resources

Data governance rules

Master data management

…

Rigid Processes for Development, Operation, and Management


What We Need is …

One universal architecture for all the data

• Transactional, external, fast (streaming), dark

One universal architecture for all formsof data usage

• From standard reporting via self-service todata science

Quick, easy, and lean access to all the data

Correct access to data


The Logical Data Warehouse Architecture

SQLdatabases

streamingdatabases

socialmedia data

productionapplication

Hadoop,NoSQL

database

website

ESBmessaging

analytics& reporting

unstructureddata

mobileApp

legacydatabase

internalportal dashboard

cloudapplications

privatedata

Logical Data Warehouse Architecture

applications

http://www.clker.com/clipart-cloud-7.html


Another View of the Logical DWA

ETLETL

Sourcesystems

Stagingarea

Analytics &reportingData

warehouse

Socialmedia data

Open data

Spreadsheets

LogicalD

ata Wareh

ou

se Arch

itecture

Big data


The Logical DWA is Metadata Driven


LogicalD

ata Wareh

ou

se Arch

itecture

ETLETL

Sourcesystems

Stagingarea

Datawarehouse

Socialmedia data

Open data

Spreadsheets

Big data

Repository


Data Virtualization Overview (1)

productionapplication website


mobileApp


Data Virtualization Server

SQLdatabases

streamingdatabases

socialmedia data

Hadoop,NoSQL

database

ESBmessaging

unstructureddatalegacy

database

cloudapplications

privatedata

applications



Data Virtualization Overview (2)

streamingdatabases

socialmedia data



mobileApp


privatedata

ODBC/SQL JDBC/SQL XML/SOAP REST/JSON XQuery MDX/DAX

JMS SQL SQL+ XSLT Hive Prop. Excel JSONCICS SOAP

applications

SQL statement

JMS message SQL statement SOAP messageData Virtualization Server

unstructureddataSQL

databasesHadoop,NoSQL

database

ESBmessaging

legacydatabase

cloudapplications



Dat

a V

irtu

aliz

atio

n S

erve

r

Virtual table pointing to source

Data consumer

Importing Source Data

Source


Dat

a V

irtu

aliz

atio

n S

erve

r

Virtual table pointing to source

Virtual table:May contain row selections, column selections, column concatenations, transformations, column and table name changes, groupings, aggregations, data cleansing, …

Data consumer

Defining Tranformations and Intregrations

Source


Layers of Virtual Tables

Enterprise data layer

Data consumptionlayer

Data sourcelayer

Data V

irtualizatio

n Server


High-Quality Data

Transactional data

External data

Fast data

Dark data

Source: Kumar Gauraw, May 2015; http://www.dataintegration.ninja/relationship-between-data-quality-and-master-data-management/


High-Quality Data at Business Speed

Integrated with logical data warehouse architecture

Data governance from the start

Lean and agile data quality

Self-service data quality

AI-driven data preparation

Good is sometimes good enough

Lean and agile master data management


Pairing Master Data and Data Virtualization


Master data Data source 1 Data source 2



mobileApp



Pairing Master Data and Data Virtualization

Prototyping

• Quick views of what golden records may look like

Consolidating multiple masters into an Enterprise Master View

• Data virtualization can combine one or more of these individual MDM systems and present one common semantically consistent “golden record” combining the most appropriate data elements from each departmental system -- all in real time

Consolidating Multiple Master Domains

• Combining siloed MDM solutions

Enriched MDM / 360 View


Consolidating Siloed MDM Solutions


Master dataCustomerFinance

Master dataCustomerMarketing

Master dataProduct



mobileApp



Future-Proofing

Your Data Strategy:For everyone

For continuous useFor instantaneous use

For analytical useFor all users

With high-quality data


Data first!


Documents

Future-Proofing Your Data Strategy - SimplicityBI · Who is using it has changed: everyone How they’re using it has changed: continuously When they’re using it has changed: instantaneously