Agile BI Development Through Automation

Tomas KratkyCEO

Manta Tools

Nigel HiggsCo-founder

Data To Value

Agenda

03

12

30

44

68

Introduction to services

We help organisations get more value from their Data

Architecture5 Core practice areas covering both business and architecture aspects of data managment

Service delivery through:

Onsite consulting

Onsite / Offsite Managed Services

Expert users of technology accelerators for bridging technology & business Data gap

Lean Data Management

Focus on reducing waste & minimising TCO

Unification of unstructured information / knowledge management & structured data management

Key mantra is minimising time spent on building solutions customers do not want

Lean Information Managemen

t

Shorter iterations

Prototyping & Minimum

Viable Products

Build-Measure-

Learn cycle

Early adopters

Cross functional

teams

Actionable metrics

Unique Value Proposition

Experience & expertise

Founders have over 40 years experience working with data

Skilled in defining data strategy and implementing data architecture,

governance and analytics solutions

Focussed on delivering business value

Align data strategy with client’s strategic goals

Work packages based on business case and ROI

Lean, agile & iterative approach

Hybrid consultancy model to scale to meet demand

Partner with innovative vendors of data tools software

We have a number of industry partnerships that allow us to hit the ground running

We also use industry leading platforms such as AWS for hosting and Tableau for Data Visualisation

Our focus is on providing customers with the most appropriate tooling to continue to make progress after initial projects have completed

Tooling & Partners

Data to Value industry partners

& platforms:

Lean Approach – Iterative Process

Maturity benchmarking

Data Profiling & Data Discovery

Harvest key metadata (apps, lineage, processes etc.)

Test rules & capture metrics

Generate risk & cost metrics

Capture quality, governance & modelling notes

Review issues using visualisations & dashboards

Prototype data solutions

Implement practical

Integrated Approach

data quality & governance

metrics

data models, glossaries & dictionaries

disparate data & metadata

data profiling & metadata discovery

ontologies controlled vocabularies

Typical Outputs

DQ Issue lists & KPIs to guide decision making

Powerful, interactive visualisations

Models, Knowledge Graphs & Glossaries to understand what

data assets you have

Dashboards articulating the data quality issues that are

holding you back

Clean, actionable & well structured data

in a variety of formats

Ready to use Prototypes & POCs

Passionate, Innovative, Lean

Lean Information Management specialists

Data to Value Ltd. 2nd Floor Elizabeth House, Waterloo, London SE1 7NQ United Kingdom

T +44 (0) 208 278 7351www.datatovalue.co.uk

info@datatovalue.co.uk

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

Enterprise vs Business Driven BI

Raju Sonawane

About Me

Twenty Six years’ experience in Business Intelligence, enterprise architecture, strategy/roadmap, design, development and project/people management.

Roles played - Head of Business Intelligence, Data Architect, BI Architect, Solution Architect, Agile Scrum Master, Project Manager and onsite/offshore Business Development Manager

Domain/Industry - Fund/Investment Management, Lloyds of London Specialist Insurance/ Reinsurance, Life Insurance and Consultancy

The content in this presentation is my opinion/view of BI. My current/previous employers may have different views.

BI Maturity in my view

Enterprise BI

Ways to deliver it

Strengths and Weaknesses

The side effect –

Homegrown UDAs

Business Driven BI

Ways to deliver it

Strategy

•Define the Enterprise BI Strategy

•Set the BI products roadmap

Enable the Platform

•Open the platform to the users

•Empower the users with self-service

Rapid Prototypes

•Promote “BI as a Service”

• Identify the high value low size use cases

•Build the rapid prototypes along with the users

Leverage User-driven BI

•Govern the user-driven BI

•Leverage the popular user-driven BI to build Enterprise BI

Ways to deliver it

Demo the existing BI capabilities

Conduct Workshops for the new use cases

Deliver the rapid prototypes with the

users

Ways to deliver it

Strengths and Weaknesses

BI Strategy

Finally…

Thank You

Oliver Cramer

The Model is the Foundation for Data Warehouse Automation

Agile BI Development Through Automation

London

DWH42

Agenda

• About me

• Understanding

• The gap

• The model

• Data Warehouse Automation

DWH42

About me

• Data Warehouse Architect

• 13 years working in the Business Intelligence area

• Since 2003 working with elementary building blocks for the Data Warehouse

• Blog www.dwh42.de Data Warehouse Automation

• Interested in the exchange of knowledge about Core Data Warehousemodeling styles

DWH42

About me

• TDWI Europe Fellow

• ANCHOR CERTIFIED MODELER Version 2014

• Certified Data Vault 2.0 Practitioner

• Coautor of „Neue Wege in der Datenmodellierung - Data Vault heißt die moderne Antwort“ in BI-Spektrum 03-2014

• Member of the Boulder BI Brain Trust

• Member of the BI-Podium Advisory Board Germany

• Responsible editor of the TDWI Germany Online Special „Data Vault“

• Organizer Data Vault Modeling and Certification, Hannoverwith Genesee Academy (CDVDM course)

DWH42

Agenda

• About me

• Understanding

• The gap

• The model

• Data Warehouse Automation

DWH42

The maturity path of understanding

• Multiple perspectives on the facts = Data Warehouse as an enablerto make your own picture of the world from existing data and information!

• One version of the facts = Data Warehouse as a recording device

• One version of the truth = Data Warehouse delivers the truth

DWH42

Data: consistency vs. availability

There is a fundamental choice to be made when data is to be 'processed':

• a choice between consistency vs. availability

or

• a choice between work upstream vs. work downstream

or

• a choice between a sustainable (long term) view vs. an opportunistic (short term) view

on data

Ronald Damhof http://prudenza.typepad.com/dwh/2015/11/there-is-a-fundamental-choice-to-be-made-when-data-is-to-be-processed-a-choice-betweenconsistency-vs-availability-or-a.html

DWH42

The confusion solution

Lars Rönnback:

"When working with information, confusion is sometimes unavoidable. To be more precise,

when the process of identification cannot give unambiguous results, such confusion arises.

... Push that problem into the future, to solve it when you find the missing pieces, while still

retaining analytic capabilities.

Simply store all the possible outcomes in advance, with different reliabilities, or store the

most likely scenario and correct it later if it was wrong.

http://www.anchormodeling.com/?page_id=360

DWH42

Main model requirements

• The model must be capable to absorb

multiple perspectives on the facts!

• The model must be capable of corrections!

DWH42

Our problem -> rendering knowledge

Dave Snowden: 7 Principles of Knowledge Management / Rendering Knowledge:

1. Knowledge can only be volunteered, it cannot be conscripted.

2. We only know what we know when we need to know it.

3. In the context of real need few people will withhold their knowledge.

4. Everything is fragmented.

5. Tolerated failure imprints learning better than success.

6. The way we know things is not the way we report we know things.

7. We always know more than we can say, and we always say more than we can write down.

http://cognitive-edge.com/blog/rendering-knowledge/

DWH42

Agenda

• About me

• Understanding

• The gap

• The model

• Data Warehouse Automation

DWH42

The gap

The big gap between modelers and business people is the language we speak!

The modelers mantra:

• We have to close the gap!

• They will never close the gap!

• They will not move in our direction!

DWH42

The logic

• We aspire to be logical modelers, to create the best logical model!

• Are the business people logical? Are they like Spock from the Starship Enterprise? Are they from the planet Vulcan?

• No, they are humans from the planet earth like we are!

DWH42

The advancement

• The model must have a fully communication orientation (in this case business speech) (is that logical modeling?)

• For this reason the model must support homonyms and synonyms!

• A synonym is a word or phrase that means exactly or nearly the same as another word or phrase in the same language.

• In linguistics, a homonym is one of a group of words that share the same pronunciation but have different meanings, whether spelled the same or not.

DWH42

Fully communication orientation-> Business model

From Quipu:

• This business model does not normally exist in any source system: it must be developed in close cooperation with the business to reflect the terms and definition of the data that the business chooses to work with. It identifies the business keys that identify the various business entities and their inter-relations. It also specifies all relevant attributes and facts related to these business entities that are required for management reporting, (predictive) analysis, etc.

http://www.datawarehousemanagement.org/

DWH42

Agenda

• About me

• Understanding

• The gap

• The model

• Data Warehouse Automation

DWH42

The model

Model requirements:

• It must have integration points.

• It must support identification.

• It must support relationships / Unit of Work relationships!

• It must support dynamic relationships.

• It must support storing attributes from different origins / integration of attributes is not necessary!

• It must categorize attributes for identification.

• It must be a model with historization capabilities. And history of history?

DWH42

The model

Model requirements:

• Support of data provenance!Data provenance refers to the ability to trace and verify the creation of data, how it has been used or moved among different databases, as well as altered throughout its lifecycle.

• It must follow standards.

• It must follow naming conventions.

• It must follow patterns.

DWH42

The model

Model requirements:

• It must be scalable.

• It must be readable and understandable!

• It must be searchable. Crawler!

• It must be partition able.

• It must be extendable.

• The possibility to model extensions without destruction of current entities!

• It must be version able.

DWH42

The model

Model requirements:

• It must support the separation of concerns!

• It must have a raw data area.

• It must have a integration area.

• It must have a rule area.

• It must have a area for sensible data.

• It must have a business area.

• It must support temporal and business perspectives.

DWH42

Agenda

• About me

• Understanding

• The gap

• The model

• Data Warehouse Automation

DWH42

Data Warehouse Automation

The big picture

All these details might make it hard to understandhow this has anything to do withautomation of Data Warehouse.

• The only two steps that can’t get automated are theinformation modeling process and the semantic mapping exercise.

• Is that statement subject to change?

• Today the rest, before applying of rules, is the domain of data warehouse automation!

• And what can be done ...!

DWH42

Data Warehouse Automation

Baseline is that the model must separate keys/identifiers, relationships andattributes / group of attributes forData Warehouse Automation.

It must be fully communication oriented, so we can close the gap to the business.

In the end we can focus on asking better questions. This is the next generation of Data Warehouse Automation.

DWH42

End

Thanks for the attention!

• Models

• Business Glossary

• Relations

• …

Implementation

• Marts

• Reports

• ETLs

• …

• Enhancement

• Consolidation

• Restructuralization

•

•

•

•

•

•

•

•

Panel discussion

Connected Data London 2016

The leading conference bringing together

the Linked and Graph Data communities

12th of July – Central London

www.connected-data.london @Connected_Data

Connected Data London

MeetUp

Join us at our informal MeetUp event. Listen to short

talks delivered by Connected dData experts and share

your ideas with like minded Connected Data fans!

7th of June – Central London

http://www.meetup.com/Connected-Data-London/ @Connected_Data

Thank you for coming!

Please fill out our survey before leaving

@DataToValue

@Manta_tools

https://www.linkedin.com/company/data-to-value-ltd

https://www.linkedin.com/company/manta-tools