Upload
danghuong
View
213
Download
0
Embed Size (px)
Citation preview
1
Chapitre de Montréal du TDWI
13 septembre 2010
Qu’est ce que TDWI ?
TDWI provides education, training, certification,
news, and research for executives and
information technology (IT) professionals
worldwide.
Founded in 1995, TDWI is the premier
educational institute for business intelligence
and data warehousing.
2
2
Mission du chapitre de Montréal
Informer objectivement la communauté sur les
meilleures pratiques BI et les tendances
émergentes en matière d’Intelligence d’Affaires.
Offrir de la formation sur mesure, promouvoir le
réseautage et le partage de connaissances en
tenant compte de la culture d’affaires
Québecoise.
3
Montreal Chapter Board
• Alain Bond, President
• Elise Lacoste, Vice President
• Sylvie Fréchette, Secretary
• Yvan Dupont, Sponsor Coordinator
• Stéphan Robitaille, Event coordinator
Bienvenu à nos petits nouveaux
• David-Marc Petit
• Frédéric Gingras
4
3
5
Merci à nos commanditaires!
66
Svp. / Please
Fermez la sonnerie de votre cellulaire /
Turn off your cell phone ringing
4
7
Ordre du jour / Agenda
08h00 - 08h10 Propos d'ouverture
Par Alain Bond
Président du Chapitre
Opening Remarks
By Alain Bond
Chapter President
08h10 - 09h15 Première partieIntroduction à l'approche de développement Agile
et discussion sur comment elle doit être adapté au
contexte des projets BI
Par Ralph Hughes
Architecte de systèmes en chef pour Ceregenics,
Inc
Part OneIntroduction to Agile development approach and
discussion on how it must be adapted for BI
projects
By Ralph Hughes
Chief Systems Architect for Ceregenics, Inc
09h15 - 09h30 Pause Break
09h30 - 10h20 Deuxième partieSolutions aux défis les plus communs qui
accablent souvent les nouvelles équipes Agile de
développement d'entrepôt de données
Par Ralph Hughes
Architecte de systèmes en chef pour Ceregenics,
Inc
Part TwoSolutions to the most common challenges that
often overwhelm new Agile warehousing teams
By Ralph Hughes
Chief Systems Architect for Ceregenics, Inc
10h20 - 10h30 Mot de la finPar Elise Lacoste
Vice-présidente du Chapitre
Closing Remarks
By Elise Lacoste
Chapter vice-president
Ralph HughesChief Systems Architect for Ceregenics, Inc
Ralph Hughes is leading business intelligence programs and
projects for Fortune 500 companies in aerospace,
telecommunications, and pharmaceuticals. He holds bachelor and
master degrees from Stanford University in computer modeling and
econometric forecasting.
A certified Scrum Master (CSM) and PMI project management
professional (PMP), he has been building data warehouses since the
early 1980s and was the primary inventor of the Agile Data
Warehousing(TM) methodology.
He lives in Denver, Colorado, skiing and fly fishing between projects.
He can be reached at [email protected],
720.951.2100.
5
Agile Delivery of BI Projects
Ralph HughesChief Systems Architect for Ceregenics, Inc
• Agile delivery and how it differs from waterfall methods?
• To what degree can Agile accelerate our BI/DW projects?
• How can fast-moving Agile BI/DW teams ensure quality?
• How to start small and grow Agile BI/DW?
• Does Agile transform other project aspects
such as estimating and architecture?
Copyright (c) 2010, Ceregenics, Inc. 10
Topics
6
Copyright (c) 2010, Ceregenics, Inc. 11
Coke vs. Pepsi
Cats vs. Dogs
Zealotry?Balanced Views Work Better
…PC vs. Mac
12
DOD-Std-2167
Applctn Rqmts
However...Waterfall *was* a Misconception!
System Reqs Design Testing OperationsCoding
Coding
Usage
“risky & invites failure”
Analysis
In the 2nd half
of Royce’s
paper
Dr. W. W. Royce, Managing the Development of Large Software Systems, 1970
…
7
Speaker’s Background
Copyright (c) 2010, Ceregenics, Inc., www.ceregenics.com 13
PharmaTelecomGovernmentOil & Gas
MA, PMP, CSM
InformaticaPL/SQL, Perl
BObj, Cognos, HyperionQlikView
TRWSonyEli Lilly Level 3Bell LabsComcastLockheed MartinAT&T BroadbandBureau of Land Management
NextelQwestSonyWolters KluwerGuidant BiomedFeed the ChildrenStanford Med Center
speed x20
cost x14
speed x4
cost x2
…
Shippable
Application
Three Major Components
Copyright (c) 2010, Ceregenics, Inc. 14
2. Agile
Development1. Agile
Requirements
Management
3. Agile
Quality
Assurance
Potentially-
Shippable
Code...and Agile QA gets you
successfully out
Requirements management gets
you successfully into a project...
Agile development delivers the code
quickly...
…
8
Scrum CoreRUP Wrapper
Agile Flywheel & Its Wrapper
Copyright (c) 2010, Ceregenics, Inc.15
16
Key Innovations of Generic Agile
• Co-located Teammates .........put business & coders in the same room
• Time-Boxed Development .....................give them three weeks
• Self-Organized Teams ............let them work as best they know
• Just-in-Time Requirements ......figuring things out as they go
• Size-Based Estimation ............................using known references
• Test-Led Development ............and a tough definition of “done”
• Automated Integration Testing ..................that’s applied daily
• PM-Light ...................with only two, paper-based tools for guidance
9
17
Self-Oranized Team: Task BoardStory by Priority
Tasks Waiting
TestsWritten
Under Development
WaitingValidation
Ready to Demo
User needs to...
2 sp
Code the...
5 hr / 2
User needs to...
8 sp
User needs to...
6 sp
Code the...
8 hr / 10Code the...
24 hr
Code the...
12 hr
Code the...
10 hr / 6
#1
#2
#3
List...
Tally...
Compare...
List...
Tally...
Compare...
Business provides
“User Stories” (est.
in story points”
Team provides
coding tasks
(est. in labor hours)
…
18
PM Lite: Burndown Chart
Demo
“Tech Debt”
…
10
Does Anyone Really Do Waterfall?
Copyright (c) 2010, Ceregenics, Inc. 19
• F4 Phantom
• Built via plan-driven process
like IT’s waterfall methods
• Introduced in 1960
with missiles only
• Supersonic dogfights
thought impossible...
• ...but missiles proved unreliable and unusable in short range
• Five years later, the fifth revision finally had cannon, smokeless engines to reduce visibility from the
ground, and leading-edge slats to improve maneuverability
• Essentially, designers followed iterative development over releases
• In IT, such long-term iterations denies the current project sponsor any benefit from lessons
learned
M
Release
DeploySystem
Test
Intgrn
TestBuild20%
Analysis &
DesignROM
Agile is Evolution – Not Revolution
Copyright (c) 2010, Ceregenics, Inc. 20
T-Shirt
ROMT-Shirt
20%
Waterfall
Agile
Dept
Arch
1 2 3 4 5
Here the project
“out drives its headlights”
M
Deploy
ReleasePotentially-Shippable
Modules
System
Test
11
21
Agile’s Iterative Approach
• Emphasizes small efforts frequently reviewed
• De-emphasizes “getting it perfect the first time”
• “Fail fast & fix quickly”
• Recursive 80/20 rule
• Maximizes learning
Full Scope
Breadth
Depth
Offers the best chance of success when facing significant unknowns
…
22
Starting a Team
Option 1: Gradual Improvement Iterations Required
Stage Fast Slow
0: Generic Scrum 1 2
1: Pipelined Delivery 1 2
2: Sized-Based Estimation & Release Plan 2 4
3: Reference Model 2 4
4: Test-Led Development 3 6
5: Requirements Mgt & Automated Testing 3 6
Total Iterations 12 24
Elapsed Time (2- and 3-week iterations) 24 – 36 wks 48 – 72 wks
…
12
23
Pipelined Delivery Squads
24
The User Story
• Index card with one, two lines
• “As a <user role>,
I want to <action>,
so that <benefit>.”
• Reminder to have a conversation
• Property of the Product Owner
• Can be moved, prioritized, partitioned quickly
– as part of the conversation
• They are the central organizing vehicle of Scrum
M
13
DWBI Adds “Developer Story”
• Typical user story
– Add the billing data for the newly acquired company to
the revenue assurance data mart
• Team creates developer stories that are actionable:
– Stage the Western-region billing data
– Scrub customers
– Add customers to the customer master data
– Scrub addresses
– Add addresses to the LOCATION master data
– Confirm that new data now flows to data mart
Copyright (c) 2010, Ceregenics, Inc. 25M
Sized-Base Estimation: Rapid & Accurate
26
“Estimating Poker” Cards
• Cards have story points on them
• Use only the Fibonacci sequence
• Use reference stories from a past sprint
• For each new story:
• Compares to previous sprints’ stories
• Coders pick cards with new estimate
• Show them simultaneously
• Discuss the outliers
• Revote till consensus
• Most stories: 3 to 5 min
14
27
Better Estimates: The Secret Reason
Demo
Here they
could see
they were
slipping
…
28
Better Estimates: Reference Model
15
29
Ordre du jour / Agenda
08h00 - 08h10 Propos d'ouverture
Par Alain Bond
Président du Chapitre
Opening Remarks
By Alain Bond
Chapter
08h10 - 09h15 Première partieIntroduction à l'approche de développement Agile
et discussion sur comment elle doit être adapté au
contexte des projets BI
Par Ralph Hughes
Architecte de systèmes en chef pour Ceregenics,
Inc
Part OneIntroduction to Agile development approach and
discussion on how it must be adapted for BI
projects
By Ralph Hughes
Chief Systems Architect for Ceregenics, Inc
09h15 - 09h30 Pause Break
09h30 - 10h20 Deuxième partieSolutions aux défis les plus communs qui
accablent souvent les nouvelles équipes Agile de
développement d'entrepôt de données
Par Ralph Hughes
Architecte de systèmes en chef pour Ceregenics,
Inc
Part TwoSolutions to the most common challenges that
often overwhelm new Agile warehousing teams
By Ralph Hughes
Chief Systems Architect for Ceregenics, Inc
10h20 - 10h30 Mot de la finPar Elise Lacoste
Vice-présidente du Chapitre
Closing Remarks
By Elise Lacoste
Chapter vice-president
30
Test-Driven Development
Story by Priority
Tasks Waiting
TestsWritten
Under Developm
ent
WaitingValidation
Ready to Demo
User needs to...
2 sp
Code the...
5 hr / 2
User needs to...
8 sp
User needs to...
6 sp
Code the...
8 hr / 10Code the...
24 hr
Code the...
12 hr
Code the...
10 hr / 6
#1
#2
#3
List...
Tally...
Compare...
List...
Tally...
Compare...
16
31
Req. Mgt: Hierarchy of DWBI Needs
Analysis
Research
Reporting (incl KPIs)
Data Access
Copyright (c) 2010, Ceregenics, Inc.
Prediction
32
Agile’s Five-Levels of Planning
Product
Product
Release
Iteration
Daily
Product Owner & Execs, 1 to 2 x / yr
Vision Statement
Product Owner & Execs, 1 to 2 x / yr
Product Roadmap (calendarized)
Product Owner & Team; 3 to 4 x / yr
Release Backlog (user stories)
Product Owner & Team; every 3 to 4
weeks, Iteration Backlog (user stories)
Team; every day
Tasks and Burndown Review
(three standard questions)
Copyright (c) 2010, Ceregenics, Inc.
Vision
Roadmap
M
…
17
33
User
Story
Requirements Mgt: A Series of Artifacts
33
Application
Use Case
Solution
Use Case
Vision
Document
Stakeholder
Request
Program
Concept
Brief
Epic
Theme
Developer
Story
Iterative
RUP / Traditional
“We going to resell
Verizon cell phones, and
we’ll need to the revenue
by month”
Problem: Business could insist IT start
at any point in these chains
“We can’t make the
partner products
program profitable if we
can’t see the gateway
data”
“Give us an estimate on
all the projects we’ve
talked about so we can
pick which ones to do
next year”
“Send over a programmer, we’ll tell
them what to build.”
“Send over a team, Mike will work
with them to create the revenue data
mart.”
“Go ask Mike how to add the
subsidiary billings to the revenue
data mart.”
“Add the 3rd party revenue flat file to
the staging layer.”
Solution: Be ready to start at any point
and retrospectively draft the RUP docs
as the project proceeds.
Mike: “I know exactly how you
should build the allocation
module.”
Copyright (c) 2010, Ceregenics, Inc.
…
34
Automated Regression / Integration Testing
Staging
Integration 3NF
Ent. Dimensional
Dept. Dimensional
Dirty Data Missing Sets Resource Failure
Initial loadIncremental loads
Nominal Data
…
18
Automating Testing: The Scenario Cycle
Copyright (c) 2010, Ceregenics, Inc. 35
36
Starting a Team
Option 1: Gradual Improvement Iterations Required
Stage Fast Slow
0: Generic Scrum 1 2
1: Pipelined Delivery 1 2
2: Sized-Based Estimation & Release Plan 2 4
3: Reference Model 2 4
4: Test-Led Development 3 6
5: Requirements Mgt & Automated Testing 3 6
Total Iterations 12 24
Elapsed Time (2- and 3-week iterations) 24 – 36 wks 48 – 72 wks
Option 2: Fast,
Test-Driven
Cultural Change
…
19
Agile vs. Really Tough DW Challenges
Situation:
• Team asked to add the newly acquired company’s
billing data to the warehouse
• Don’t know the data’s structure, attributes, or quality
• Management wants to know how long will it take and
how much is it going to cost...
• ...so they can decide what degree of integration they’ll
sponsor
38
Bonnie’s Maxim
There are no silver bullets...
Copyright (c) 2010, Ceregenics, Inc. 39
...but monsters are
real.
…
20
Agile vs. Really Tough DW Challenges
Response:
• Agile does’t mean you can’t use all your smarts and tools!
– *Do* profile the data, calculate a ROM estimate, etc.
• “Don’t know the data’s structure, attributes, or quality....”
– Agile is the best way to proceed when facing a large unknown
– We get started after one-fifth as much analysis (i.e., 80/20 rule)
– We visualize & demo data as we go, for developer and business
– The problems will bubble up to the surface much faster
• “Management wants to know how long will it take and how much”
– Give us two iterations against the real problem, and we’ll give you
an estimate based upon real knowledge, not speculation
40
Agile vs. Really Tough DW Challenges
Summary: If it’s going to be “mission impossible”
...tackle it with Agile,
...because disaster will happen in iteration 1 or 2
...rather than leaving all the problems to suddenly appear
in user acceptance testing.
Agile can’t boil the ocean any better than
waterfall can, but it fail to do it faster.
41
…
21
42
Agility through Back Filling the Architecture
Staging
Historical
Archive Integration Dimensional OLAP Department
Access
Trends
360 Vision
Dash boards
1:
2:
3:
4:
5:
Single Version of the Truth
End User
1% of data
5% of data
10% of data
25% of data
Full data
Copyright (c) 2010, Ceregenics, Inc.,
www.ceregenics.com
…
(c) Ceregenics 2009 43
Integration
Source
Agile Trims Standard DW Architecture
Staging
Presentation
Front End Tools
ETL = $$$
ETL = $$$
ETL = $$$
Source
Source Archive
Integration
Presentation
Front End Tools
Config
Config
Config
Change Data Capture
Data Warehouse Appliances
Associative Query Engines
ConfigXX
XX
ETL
ETL
ETL
Maximize the
objects not built
…
22
Advanced Topics
• Agile & Project Risk
• Agile Resource Planning
• Agile Architecture?
• Bidding an Agile Project
Copyright (c) 2010, Ceregenics, Inc. 44M
Agile & Project Risk
Risks
– Poor business requirements
– Misunderstood requirements
– Needless requirements
– Institutionalized scope creep
– Under-employed developers
– Untested product
Copyright (c) 2010, Ceregenics, Inc. 45
via
Requirements Management
pipelined early starts & task board
iterative & automated testing
M
Mitigation
…
23
Agile & Project Risk
Further Risks
• Rework
– ETL sensitive to data model changes
– Refactoring for re-usability
• Team disruption
– Re-assignment and production support
• Geographically-remote teammates
– Possible if performed carefully
Copyright (c) 2010, Ceregenics, Inc. 46M
Team leaders’ docs developed
during iterations -1 and 0:
• Vision Doc
• Initial Data Model
• Initial Ref. Model
Mitigation
…
Resource Planning
Advantages of Agile
• Size-based estimation fast & more accurate
• 20% LOE estimates easier to derive
• Faster to spin-up projects & minimize bench time
• Staff planning becomes “We run three teams, that’s what we do”
• Agile is fast to train
• Quick, thorough, and standard means to manage staff augmentation
Disadvantages of Agile
• Requires “agile” temperament, esp. in leadership positions
• Greatest velocity from consistent team rosters
• Emphasizes co-located teams
Copyright (c) 2010, Ceregenics, Inc. 47M
24
Agile Architecture?
Concerns
• “Incremental optimization” can equal “muddling through”
• “How to avoid an Apollo 13-type situation?”
Mititgations
• Wrapper process includes requirements management
• Architecture comes from departmental standards
• New & smaller programs can externally acquire an architecture
• Reserve 20% of team bandwidth for non-functional requirements
Copyright (c) 2010, Ceregenics, Inc. 48M
Bidding an Agile Project
Challenges
• Need 2 to 3 iterations to establish a velocity
• Product owner in command of backlog
Possible solutions
• Two-part bids: “Establish Velocity” and “Remaining Deliverables”
• Reference Architecture
• “Reference” Agile team to backlog and story point a new project
• Analyze as waterfall and reduce proportionally for Agile approach
• Requirements management to constrain backlog churn
• Any new requirement affecting scope, budget, or duration CR
• Customer’s ability to conclude early can lead a fixed-price project to T&M
...puts you on the same side of the table with them
Copyright (c) 2010, Ceregenics, Inc. 49M
25
Summary
• High-level view of how Agile DWBI works
• How Agile DWBI is reducing time-to-market
• Estimating and quality get better under Agile
• Start-up path for a DWBI team or shop
Copyright (c) 2010, Ceregenics, Inc. 50
51
For More Information
Available via
Amazon.com and Barnes & Noble
Ralph Hughes
Ceregenics, Inc.
999 18th St., Ste 3000,
Denver Co 80202
720.951.2100
26
52
Questions
Prochaine rencontre
Nous sommes heureux d’annoncer que notre
prochaine rencontre sera
53
Au programme:
À venir...
27
54
Merci à nos commanditaires!
55
TDWI Membership Offer
• 10% discount on TDWI Membership
– For TDWI Chapter attendees; Limited time only
– You can renew or extend your existing TDWI membership
• Benefits
– Discounts on conferences, seminars, certification, books
– Valuable reports and publications from TDWI Research
– Web archive of research
– Inquiry Service for Team Members
• Individual and team memberships options
• Join today!
28
57
Environnement
Nous récupérerons les enveloppes de plastique
We recycle plastic name tags
58
Annonces
Informez-vous à l’avance et nous pourrons
annoncer votre événement BI.
Let us know in advance and we
could announce your BI event.
29
59
La présentation
La présentation sera envoyée
aux gens présents seulement
The presentation will only be sent to poeple who
are present today.
60
Mot de la fin / Parting words
Commentaires & suggestions à
Send comments and suggestions to
30
61
Merci !
Thank you!