Best Practices: Datawarehouse Automation Conference September 20, 2012 - Amsterdam

Datawarehouse AutomationBest practices and customer cases

20 september 2012Amsterdam

Erik [email protected]@erikfransen

mailto:[email protected]

Best practices & customer cases2

Agenda

• BI Challenges of today• BI Ability Model & DWH Automation• Best practices• Cases: Rotterdam & KAS BANK• Q&A


Centennium BI expertisehuis• Independent knowledge partner• Provides clients with the right skills, at the right

time and the right way to maximize BI results• Consultancy, ad-interim support, project

implementation and training services• We take or share responsibility for the

execution and management and support of BI and DWH projects

• We offer an extensive portfolio of courses and training services: www.bi-opleidingen.nl

• We provide our customers with the knowledge and practical insights required to be(come) self-sufficient in maintaining and expanding their BI-environments


What are (y)our biggest BI-challenges today?

• Create more business value• Empower users• Lower the overall cost• Deliver high quality BI products• Reduce complexity• Organize Business Intelligence to

become more effective

How can DWH automation contribute?


Ability to Benefit

Ability to Specify

Ability to Implemen

t

Ability to Execute

DEMAND

SUPPLY

DWH Automation – in perspective

BI Ability Model


Automation – in perspective

• Supports your ability to execute– Cuts complexity and resources

• Requires a strategy to change focus from coding to (data) modeling

But….• You’re the solution – Automation is your

instrument…


The solution?

• Automation vs. BI Professionals or….• The tool is the tool, the methodology is

the methodology, etc… • Team up to optimize our profession in

helping organizations to reach ‘infinity and beyond…’

Methodologies Automation tools

ProfessionalsCommunity


Back to the Challenges

• How does DWH automation help face the challenges?

• Directly and indirectly.

• In short… DWH automation is great (!) if handled by a skilled and knowledgeable professional (you!)


CDM: set of best practices• Best practice methodology for BI and DWH

automation• Create datawarehouse and BI products fast, with high

and constant quality and low cost• CDM includes:

– Tooling (free, open source or licensed)– Modeling paradigms like, DV, 3NF, DM– Quality control mechanism– Agile development– Knowledge partnership, training, coaching

• Quality control mechanism: extensive checklists and documentation

• Knowledge transfer by training, certification and learning on-the-job


CDM

Structure

Organize

Knowledge Partnership

Model PresentGenerat

e Benefit


Your datawarehouse is like a diner in a 3 star restaurant…

Create cook book

Manage the kitchen

Suppliers

Select Ingredient

sServeCook Eat


Knowledge partnership (suppliers)

• Goal: self-supporting clients• On-the-job coaching, learning by doing• Training

– BI fundamentals– Data Vault certification – Dimensional modelling– Tool training (partners)

• Centennium supports it’s customers by (management) consultancy, assessments, projects, training and sourcing


Structure (create cookbook)

• Establish a common vision on DWH and BI and the role of automation

• Identify needs, pain, benefits, goals• Create a roadmap: what and how?• Develop teams, knowledge, new roles:

prepare organization for automation• Design automation architecture • Create short list tools• Select agile approach


Model (select ingredients)• Information & requirements analysis

– Business needs like KPIs, reports, cubes– Identify semantic gaps, business rules– MDM, MTM, Reference data– Business keys

• Analyze & model source extraction to support automation

• Model the staging, datawarehouse, data marts, meta layers, cubes

• Select tool(s) for generation– DWH Automation tools– Combination of “classic” ETL tools and automation

tools

• Should fit the structure!


Generate (Cook)

• Create system setup with respect to automation architecture

• Develop test scenarios– Generic testing of automation process

(completeness and correctness)– Test compliance to architecture

• Generate components and objects– Focus on understanding data, data

modeling & business rules, not on coding

• Test and implement


Present (serve)

• Deliver information products to users, fast and of high quality

• Automation leverages BI self-service– Model and generate cubes– Model and generate business rules – Adapt quickly to changing source data and

information needs

– Reduced technical complexity– Speed up of the overall BI process


Benefit (eat)

• The business users can focus on creating value adding information products

• Constant and predictable quality• Short time to market• IT can quickly adapt to changing

business needs/focus• More time and resources for creating

value-adding BI• In the end: lower cost, higher value• But…


Organize (manage the kitchen)

• Automation may have high impact on existing datawarehouse and BI teams– Elimination of tools, data manipulation processes

and people… (cutting complexity and resources)– Also: new roles like source analyst, data model

specialist, automation architect are introduced– Resistance by traditional vendors and suppliers

• Automation can create high business value, but not on it’s own– A knowledgeable team is essential

• We believe in self service: organize knowledge transfer a.s.a.p. and coach teams to be self supporting


Rotterdam• Challenge: service team as a central and flexible point

for management information delivery for employees and partners of Rotterdam

• Structure:– As a knowledge partner we combined Rotterdam’s knowledge

on Oracle eBS with DWH automation– Automation architecture “forces” Rotterdam to comply to the

rules

• Model & generate– DWH automation optimized for Oracle eBS– Reusable methods for other eBS customers

• Benefit– Cuts complexity and lowers licensing & consultancy fees

significantly– DWH is now of strategic importance and acts as a central data

hub for Rotterdam– Self service for users


• Challenge– Deliver integrated information instead of stovepipe data– Minimize development cost, maximize business value– Make KAS BANK self supporting as much as possible

• Structure– Create a shared view on datawarehousing and automation benefits

and proof it– Data Vault Methodology, near real-time and high volume– Strong focus on team training and coaching of architects

• Model & generate– Re-usable extraction of complex Oracle and Adabas data stores– Business driven Data Vault and Dimensional models– Generated data integration, data distribution, EDW code, without

ETL tools– Transparent and self supporting

• Benefit– Significantly shorter time to market of information products– 100% reliability, auditability and predictability– Business is eager for more– Next step: introduce self-service at business level


Automation – do’s

• Do’s– Define an automation vision and strategy

• end-to-end or step-by-step – Take your time and involve all stakeholders– Explain concepts, (business) benefits and

potential risks– Consider a two-step approach: pilot -

project– Align and train the development- and

maintenance teams


Automation –don ‘ts

• Don’ts– Deviate from the Structure– Generate or automate the DWH as a goal,

not a means to an end– Underestimate the need for presenting

and benefiting from information…

The proof of the pudding is in the eating!


Experience with CDM and DWH Automation

CDM is an evolving set of best practices Introducing additional modelling approaches We are partnering with DWH automation vendors Research topic: generating datawarehouse models directly from business process models

Best practices & customer cases

Next data vault certification seminar: November 1-2, 2012 Amsterdam

www.data-vault.nlwww. geneseeacademy.com

24

25 october, the HagueCrash Course Datawarehouse

Automationwww.bi-opleidingen.nl


ABOUT CENTENNIUM

26

Centennium BI expertisehuis houses all the experts under one roof, hereby offering all knowledge and expertise to address the complex business intelligence issues facing our clients today

Services overview:• Consultancy• Projects • Resourcing • Education

Some of our clients: Woonbron, Albron, NZa, CAK, OBR, Vopak, several Dutch Municipalities, Aegon, Nutreco, TNO, Genzyme, Tata Steel, KPN, DELTA, IKEA, Accell, TomTom, KAS BANK, LeasePlan, Brabant Water

Facts and figures:• Founded: 1998• 45+ business intelligence

consultants

Core values:• Human Capital• In close collaboration• Objective and Independent

Expertise:• Business intelligence• Strategic, tactic and operational• Vision based on “effective BI”

27

Centennium BI expertisehuisLange Voorhout 432514 EC 's-Gravenhage Telefoon 070 31 20 370Fax 070 31 20 371URL www.centennium.nl

www.bi-opleidingen.nlwww.data-vault.nl

Technology

Best Practices: Datawarehouse Automation Conference September 20, 2012 - Amsterdam