AGILE Does it work with Data Warehouses?

What is Agile BI?

“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more”

The Agile Manifesto - 2001

Source: www.agilemanifesto.org

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

4. Working software is the primary measure of progress.

12 Agile Principles – Process

5. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

6. Business people and developers must work together daily throughout the project.

7. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

12 Agile Principles - People

8. The most efficient and effective method of conveying information is face-to-face conversation.

9. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour.

10. Continuous attention to technical excellence and good design enhances agility.

11. Simplicity--the art of maximizing the amount of work not done--is essential.

12. The best architectures, requirements, and designs emerge from self-organizing teams.

12 Agile Principles - Other

What people say about Agile

For Agile: Increases delivery

speed Decreases wasted

effort Deals with change Fewer costly failures More value delivered,

faster and earlier

Against Agile No documentation Lots of re-work No time for

architecture or design Cowboy delivery No “Big Picture” Makes mistakes due

to lake of planning Doesn’t work in BI

SuncorpAn Agile Case Study

Trained all IT staff and selected users in Agile delivery methodology

Removed Waterfall-based project funding Showed commitment from CIO down Developed communities of practice in Agile Connected with partners better than us in

Agile so that we could learn from them Invested in infrastructure to support Agile Built values and a culture around Agile

What Suncorp did for Agile

What Agile means at Suncorp

Trust, Innovation, Accountability, Courage, & Honesty

Collaboration, Flexibility, Team Work, Business Value, Simplicity, & Speed

Stand-ups, Show-Cases, Retrospectives, Pair Programming, Adaptive Planning, Sustainable Pace, Auto Testing, Test Driven Development, Continuous Integration

Practices

Principles

Values

CONCEPT INITIATE DELIVER DEPLOY

5% 5%85%5%

Suncorp’s Agile BI System Automated Testing Continuous Integration Agile-friendly ETL tools User self-development

areas Fortnightly Iterations

with showcases Multi-location teams Burn-up charts for

progress monitoring

Fortnightly IterationsWeek 1 Week 2

Iteration Planning Mtg

Joint Development

UAT and Signoff

ShowCase and Retro

Backlog Expansion for next Iteration

Details of the CI/Auto Testing

Package created

Committed to

Subversion Tests

Data

If OK Move to

Devmaster

Apply Tests

Install ETL

Package(s

)

CI Devmaster

Schema

1. When ETL code is changed and code packages are created, they are checked in to Subversion along with tests and CSV test data.

2. Hudson CI Server monitors Subversion and initiates whenever changes are detected.

3. Hudson starts a build comprising installation of new ETL package(s) to CI Schema and runs Concordion for automated testing

4. If the build is successful, Hudson installs the ETL package(s) into the DevMaster schema for use.

5. Hudson lights the Red or Green traffic lights to demonstrate results

ETL

Some documentation is needed, but not too much.

Not all developers like doing testing! Need an Iteration 0 to set up architecture

and platform first. It’s all or nothing. Using waterfall

underneath doesn’t work, as per the Half-Arsed Agile Manifesto

Traps in Agile BI

We have heard about new ways of developing software by paying consultants and reading Gartner reports. Through this we have been told to value: Individuals and interactions over processes and tools and

we have mandatory processes and tools to control how those individuals interact

Working software over comprehensive documentation as long as that software is comprehensively documented

Customer collaboration over contract negotiation within the boundaries of strict contracts, of course, and subject to rigorous change control

Responding to change over following a plan provided a detailed plan is in place to respond to the change, and it is followed precisely

That is, while the items on the left sound nice in theory, we’re an enterprise company, and there’s no way we’re letting go of the items on the right

Half Arsed Agile Manifesto

Source: www.halfarsedagilemanifesto.org

In Iteration 0, do some initial architecture envisioning

Model the details just in time  Prove the architecture early Focus on usage Organize your work by requirements in priority order Ensure active stakeholder participation Take an evolutionary approach Test throughout the lifecycle Involve operations and support people early Adopt common development standards

Tips for Agile BI

Source: adjusted from http://www.agiledata.org/essays/dataWarehousingBestPractices.html

Is Agile BI the Holy Grail?

Answer: NO

What people say about Agile

For Agile:

Increases delivery speed

Decreases wasted effort

Deals with change

Fewer costly failures

More value delivered, faster and earlier

Against Agile

No documentation

Lots of re-work

No time for architecture or design

Cowboy delivery

No “Big Picture”

Makes mistakes due to lake of planning

Doesn’t work in BI

½½½½

Questions?