How We Became World Leaders in Agile Data Warehousing Em Campbell-Pretty

General Manger. Strategic Delivery (EDW)

Business Intelligence CoE, Telstra

@PrettyAgile

The Enterprise Data Warehouse

80 TB of business data

28 Teradata nodes 3 TB of new data per month

350 SQL users 47 business systems feeding into

the EDW

400 million records loaded daily 13 billion records extracted daily

43 billion records in largest table

4 tables with over 35 billion records

124 different client apps

6.8 million queries daily 16K tables, 28K views

and 2K macros 7K jobs running daily

21K processes

The Team

Average delivery cycle time down from 12 month to 3 months

Frequency of delivery increased from quarterly to fortnightly

Cost to deliver down 50%

95% decrease in product defects

100% projects delivered on time and on budget

Happy project sponsors

Happy teams

The Results

WHERE SHOULD YOU START?

START WHERE YOU ARE

http://www.flickr.com/photos/joelogon/4613630072/

Read, Understand, Adapt & Apply

http://www.flickr.com/photos/duluoz_cats/5300924038/

Wide Pipes & Batch Patterns

Agile Pilot

5 “Wagile” Teams

Image source: http://www.vectorloft.com/img/purple-person.jpg

www.scaledagileframework.com

17PRESENTATION TITLE | PRESENTER NAME | DATE |

Deployment

Services

Development Services

Pipeline

Services

Structuring the Agile Release Train

18PRESENTATION TITLE | PRESENTER NAME | DATE |

Agile at the Program Level

MAKE THE INVISIBLE INVENTORY VISIBLE

Reduce Batch Size by Breaking Down the Work

21PRESENTATION TITLE | PRESENTER NAME | DATE |

Agile Feature Teams

Scrum Provides Cadence & Fast Feedback

Image Siource: http://innolution.com/uploads/misc/Essential_Scrum_Chapter_2.pdf

Smaller, Multi-skilled Teams Counter Balance Variability

Project Manager

Product Owner

Business Analyst

Change Lead Technical

Lead

Logical Data Modeller

Physical Data Modeller

Physical Data Modeller

Vender Project Manger

On Site ETL On Site ETL

On Site ETL Off shore ETL

Off shore ETL Off shore ETL

DBA

Business Intelligence

Test Lead

Test Analyst

Enterprise Architect

Data SME

Scrum Master

Project Manger

Scrum Master

Technical Lead

Test Lead Developer Developer Developer Developer Developer

As is

As was

EPIC OWNER

FEATURE OWNER

PRODUCT OWNER

Scale the Product Owner to Balance Centralised & Decentralised Control

Extreme programming improves quality

Extreme Programming Improves Quality

Limiting Work in Process

(WIP) Improves Flow

27PRESENTATION TITLE | PRESENTER NAME | DATE |

The System Team

Build the Infrastructure to Enable Economically Viable Smaller Batches

29PRESENTATION TITLE | PRESENTER NAME | DATE |

The Agile Release Train

Cocktail Hour Proves a Daily Communication Cadence

9:00am 9:15am 930am

9:45am

Unity Day Leads to One Team Culture

31

A Little Taste of Our Culture

http://www.prettyagile.com/2013/05/the-power-of-haka.html

The Software House of Lean

Respect for

People

Product

Development

Flow

Kaizen

1. Take an economic view

2. Actively manage queues

3. Understand and exploit variability

4. Reduce batch sizes

5. Apply WIP constraints

6. Control flow under uncertainty: cadence and synchronization

7. Get feedback as fast as possible

8. Decentralize control

The Principles of Product Development Flow

Respect for People: The Net Promoter System (NPS)

Source: http://www.netpromotersystem.com/

Net Promoter Score

Kaizen Mindset: Continuous Improvement and Learning Culture

3

Source: https://dl.dropboxusercontent.com/u/1018963/Articles/SpotifyScaling.pdf

The Foundation is Servant Leadership

“This is probably the most well-scaled agile data warehousing group I’ve ever observed, and they are still getting better” Ken Collier

Director of Agile Analytics at Thoughtworks & author of Agile Analytics

Source: http://theagilist.com/2013/09/03/scaling-agile-data-warehousing/

WHERE SHOULD YOU START?

START WHERE YOU ARE

http://www.flickr.com/photos/joelogon/4613630072/

"Adapt what is useful, reject what is useless, and add what is specifically your own."

- BRUCE LEE

Check out my blog: www.prettyagile.com

• A Perspective on the Scaled Agile Framework

• What Happens to Project Managers When You Implement SAFe?

• The Power of Haka

• Book Clubs at Work – Are You Serious?

• Leading Through Vulnerability

• The "Bubble Up" Approach to Scaling Retrospectives

• Bookshelf

Want to Know More About the EDW Agile Release Train?

PARTNERS Mobile App

InfoHub Kiosks

teradata-partners.com

Twitter: @

Email:

Emily.Campbell-Pretty@team.tesltra.com

PrettyAgile