Improving ROI and Success Rate of Your Business Intelligence Project

Improving ROI and success rate of your Business Intelligence project using an AGILE iterative and incremental approach By Martin Proulx and Dominic Danis

About the authors Over the years and currently as Vice-President, Agile Business Intelligence at Pyxis Technologies, Martin Proulx has successfully participated in several projects with companies from the pharmaceutical, retail, financial, and manufacturing sectors. Martin holds a Master's degree in management sciences and a Bachelor of Business Administration from HEC Montreal. He cumulates over 15 years of experience in: software development project management, business process optimization, and dashboards and key performance indicators (KPIs). Dominic Danis cumulates over 10 years of experience in software development and BI project management. In an effort to use a better approach to software development Dominic has been using Agile and Scrum since 2005. He is a Certified Scrum Master and he successfully applies this approach to Business Intelligence projects. Dominic is currently Principal, Agile Business Intelligence at Pyxis Technologies. About Pyxis Technologies Pyxis is the name of the constellation of the compass and as such Pyxis acts as beacon for companies seeking to improve their ways of doing software development. Pyxis was founded in 2000 as the result of reflection of engineers in computer science questioning the traditional approaches to software development.

Executive Summary Business intelligence (BI) applications are increasingly popular and organizations of all sizes are seeing the benefits. As a consequence, BI applications are no longer limited to senior executive but are increasingly deployed throughout departments and levels. The timely completion and successful implementation of BI projects provides a competitive advantage to the adopting organizations. Unfortunately, there are more failures and disappointments than successes. It is generally accepted that other pre-requisites such as: ensuring executive support, aligning business and IT strategies or selecting the appropriate technology platform are critical to the success of BI projects. In addition to these requirements, we believe that an AGILE iterative and incremental development approach combined with light weight project management principles, such as SCRUM, greatly improve this track record. In addition, since the development approach and project management techniques are mostly under the control of the information technology (IT) executive, they can more easily be altered in order to deliver better results to the organization. Our approach relies on simple and straightforward principles. Through short cycles of definition, planning, building, presenting, and retrospecting, the development team iteratively delivers value and incrementally adds components to each other to achieve a complete BI application. As long as the development team focuses on highest value KPIs, building them from beginning to end incrementally and routinely demonstrating their work to decision makers for feedback, they will remain focused. Our objective is to increase the ROI of the BI initiatives by:

Reducing the initial investment required to launch a BI project by quickly delivering tangible components to the business users instead of heavily investing upfront in architecture and modelling costs.

Accelerating the availability of the information by using an

Copyright © 2009 Pyxis Technologies www.pyxis-tech.com

2 Improving ROI and success rate of your business intelligence project through an iterative and incremental approach

Their questioning led them to adopt new approaches to software development project management and to improve their processes and practices.

incremental approach to present high level indicators at first and refining the requirements towards more granular information.

Delivering the right solutions to the business users by working iteratively with them to fully understand the requirements.

Prioritizing the work so the requirements with the highest value and ROI are addressed first, thereby continuously delivering values to the organization.

Track record of BI projects There was a running gag in an organization we were working with that every Business Intelligence (BI)1

project required 3 years to complete and would cost over $3 Million. This running gag demonstrated an

unfortunate situation – most BI projects are expensive, time and resource consuming and have a low

rate of success. Along these lines, very few people would debate that traditional BI projects require

heavy investments and result in long delays before providing value to the organization.

Much has already been said about the importance of BI applications in organizations. One of the most

obvious and frequently repeated benefits of BI applications is the ability to increase the quality and

timeliness of the business decisions being made. This is increasingly true as BI applications are being

democratized and are now embedded in the decision making process at every level of the organization

and across multiple departments.

In addition, decision makers constantly need an in-depth understanding of their business’ strengths and

weaknesses and this is especially true during difficult economic times. The availability of timely and

accurate information can help business leaders make the right decisions. Along these lines, the survey of

over 1,500 CIOs conducted by Gartner EXP2 shows that despite predicted flat IT budget growth in 2009,

BI projects remain their number one technology priority.

There isn’t much need to emphasize further the value of early diagnosis and the implementation of

timely solutions but in a changing business environment the business needs evolve too quickly for

sequential waterfall development methodologies such as the one presented in the following diagram.

1 Within the context of this white paper and for simplicity, we combine Data Warehouse (DW) projects with

Business Intelligence (BI) projects. 2 http://www.gartner.com/it/page.jsp?id=855612

http://www.gartner.com/it/page.jsp?id=855612



Case in point, it is estimated that 60%3 of BI projects end in abandonment or failure and market data

confirms that out of the 3 standard project dimensions – time, resources, and scope – timelines and

resources typically exceed the initial plan while scope consistently fall short of the original expectations

and requirements. As such, an approach that delivers value early in the project while remaining aligned

with the business priorities is recommended.

In addition, estimates show that no more than 20%4 of business users actually use their BI applications

proactively and that a staggering 64%5 of systems functionalities are rarely or never used. This

regrettable situation compelled us to look for an alternate and improved approach to ensure BI projects

have greater success rate. More specifically that BI projects deliver the features required by business

users (scope) in a timely fashion (time) while controlling expenses (budget).

Our intent is neither to rehash the unfortunate track record of BI projects nor to highlight the most

dramatic failures but to propose a different approach to the development of BI projects.

This is a tall order but with the increasing popularity of AGILE iterative and incremental approaches and

with repeated success in other specialities; we know using such an approach in the development of BI

projects leads to much better outcomes.

An iterative and incremental approach for BI projects The unimpressive track record in the development of BI applications led us to conceive an improved

approach to reduce the initial investment to launch a BI project and to modify the requirement

3 Business Intelligence Roadmap, Moss and Atre, 2003

4 Business Intelligence Summit by Gartner, 2008

5 Standish Group Study Reported at XP2002 by Jim Johnson



gathering phase in order to deliver value much more quickly and at lower cost. In addition, since our

incremental development approach remains aligned with the business priorities, our method allows

business users to further develop the knowledge of their business and better define their expectations

so that the end result meets their needs.

Although our approach requires a substantial paradigm shift for most people having experienced a

traditional waterfall approach for BI projects, the notions we propose are simple, logical and

straightforward. The proposed approach relies on AGILE principles and on SCRUM practices that have

been demonstrated to be successful in other types of initiatives time and time again. Our objective is to

present them in a concise manner in this white paper.

The proposed approach to BI development can be summarized by the following principles:

Segment the development of components into small iterations of 1 to 4 weeks and frequently

deliver completed objects (key performance indicators, metrics, reports, dashboards, cubes,

etc.) at the end of these iterations;

Repeat the iterative process and use increments to add components to the BI architecture and

infrastructure;

Prioritize the development of components that will deliver the highest value to the organization

first;

Use a “slicing approach” to go from source systems to presentation layer;

Avoid analysis-paralysis by using a non-superfluous approach in documentation, architecture,

modeling, project management and other overhead activities;

Learn from each of the iteration and adapt the process accordingly to increase efficiency.

Our approach promotes a process that encourages frequent inspection and adaptation, a set of

engineering best practices that allow for rapid delivery of high-quality software, and a business

approach that aligns development with customer needs and company goals. With this approach, the

project delivers small increments with minimal planning, rather than long-term planning which impact

the organization’s ability to react to changing needs.

As opposed to the traditional approach to building a BI application which is to plan the project, define

the requirements, prepare the architecture, develop, test, move to production in a sequential manner,

we use an iterative approach during which we build layers of the entire data warehouse through

iterations. For example, instead of spending time on defining high level requirements, working with the

end users to understand their business and the metrics used to successfully manage it, our objective is

to use an iterative approach and work backward starting with the required performance indicators back

to the source systems.



As can be seen in this diagram, the intent is not to divide the project into phases such as: data extraction

from the source systems, data cleansing and standardisation, Extract-Transform-Load (ETL) to a

centralized repository and so forth. Instead of the traditional approach, we use a “slicing approach”

approach where each iteration produces complete components developed from source system to the

presentation layer.

Success requires a paradigm shift As we mentioned, the successful implementation of an AGILE iterative and incremental approach

requires an important paradigm shift in the way the project team operates. Although many other

variables may be impacted, we focus our efforts on five key areas: simplify requirements gathering,

become more flexible, reduce overhead, focus on value, and deliver on expectations.

1. Simplify requirements gathering Our approach focuses on eliminating the assumption that decision makers clearly know what

information they need to perform their role and can easily describe what information they require

up front at the beginning of the project. By spreading the requirements gathering throughout the

duration of the BI project, the development team reduces and spreads the amount of stress

imposed on business users and is more likely to obtain meaningful requirements.

Based on the principle that end users may only know a subset of all the metrics they need to

successfully manage their business, we need to build the knowledge and expertise based on small

and specific known components at the beginning and evolve to more complex metrics once the

users understand the process.



2. Become more flexible In an ever changing business context, one’s ability to anticipate and clearly document their need for

the next 12, 18, or 24 months seems absurd. Very few people would have predicted the current

financial context 6 or 12 months ago. Unfortunately, traditional development approach still heavily

relies on such a process to launch and develop a BI project. By allowing the project team to react to

changing requirements due to an ever changing business environment throughout the life of the

project, they increase the probability that the resulting cubes, reports, and dashboards get used

while ensuring the project team delivers value throughout the duration of the project.

The emphasis is on obtaining the smallest workable piece of functionality to deliver business value

early, and continually improving it and adding further functionality throughout the life of the

project.

3. Reduce overhead The cone of uncertainty presents the reduction in the uncertainty of a project as it progresses in time. For instance, the variability of the outcome is much more difficult to predict at the early stages of the project than it is toward the end. As such, much less effort should be invested in planning, architecting, modelling, documenting and defining requirements in the early stages of the project as they are likely to change. Consequently, the level of details and the quality of the knowledge should increase as the project progresses.



4. Focus on value In contrast to a traditional approach which sequentially progresses through phases (i.e.

requirements definition, analysis, design, coding, and testing) in a strict pre-planned sequence which

measures progress in terms of delivered artefacts (requirement specifications, design documents,

test plans, code reviews), our approach focuses on completed useful software components.

Although these artefacts have value for the project team, we feel too much time and effort is

invested in these intermediate deliveries instead of working software. The recommended approach

does not completely eliminate these artefacts but reduces them to a non-superfluous level in order

to remain focused on the key outcome which is to deliver value.

In addition, our proposed approach focuses on value by increasing the ROI of BI components being

developed and deliver higher value components earlier in the process and halt the project when the

cost of developing exceeds the expected value.

5. Deliver on expectations By presenting completed deliverables frequently, such as KPIs, metrics, reports or cubes to the business users at the end of each iteration, this allows any potential issues with the quality and accuracy of the data to surface and be addressed.

Four levels of an Agile BI project Our approach to the development of a BI project is divided into 4 levels: Program, Project, Release and

Sprint. Contrary to sequential development, these levels are not consecutive but encompassing in

nature where a program is composed of projects which are themselves composed of releases and each

release is divided into sprints.

BI projects are often seen as large and complex endeavours and in order to maintain an enterprise wide

view and ensure constant alignment, they are grouped under an overall program.

At the project level, activities such as project definition, staffing, and budget definition are executed.

Although important, our intent is not to focus on the initiation of the project but it is critical to

remember that activities during the project level remain at a high level due to the amount of uncertainty

still surrounding the project.

A release is composed of an agreed upon number of sprints during which components of the BI

application will be completed. For example a release could cover the need of a specific business area

such as sales, marketing, operations or a business process such as order fulfilment, shipping, or client

satisfaction.



Once the release has been defined at a high level, it is then divided into smaller components – called

sprints. Based on the size and experience of the development team, a sprint can be anywhere between

1 to 4 weeks.

With a traditional development approach, each task must be completed before the development team

can move to the next phase whereas with an iterative and incremental approach, the development

team delivers small components frequently.

Five steps repeated throughout the four levels The process is iterative and incremental at the four levels of the project with a major distinction in the

amount of details being analyzed. While the project team will remain at a high level during the project

cycle, they will look for more details during the release cycle and very granular details at the sprint cycle.

For example, at the project cycle the business user is expected to explain that the project team will need

to provide performance indicators for his consulting department. Once the team cycles through the

release cycle, the business user would explain that he is looking for utilization rate for his consultants

and at the sprint level he would provide information as to how the utilisation rate is calculated.

While the traditional approach to a BI project calls for much of the efforts to be invested in planning and

analysis, an incremental and iterative approach prefers to deliver small components frequently in order

to obtain feedback and learn from the exercise.

Define The objective of the first step of our approach is to get familiar with the project, the main players, and

the expected outcome, as well as, understand the priorities. During these sessions, the project team

interviews the key business users to gather their requirements and refine their understanding of the

business challenges. From a project perspective, a business representative is assigned to the initiative



with the objective of coordinating the business users involved with the project, due to the fact that BI

typically impacts various departments.

During the first meetings the conversation with the business users is around their business issues,

concerns and priorities. The objective is to have a good understanding of the various dimensions of their

business and key members that will support the main representative for the definition of performance

indicators for their department. The outcome of the definition stage is to draft a project charter that

includes the following elements: vision for the project, objectives, assumptions, success factors, success

criteria, roles and responsibilities, priority matrix (scope, timeline, and budget), risks, and mitigation. In

addition to a project charter, the planning stage allows the project team to start populating a product

backlog. During the sessions, the team’s objective is to derive stories that will later be prioritized.

Through a well defined process, the team executes a requirement definition phase through stories6. We

typically use the following structure to derive the various stories: “As [role], I need [metric or ratio] by

[geography, time period, etc.] in order to [business decision or business value]”. For example, the

business representative speaking on behalf of his director of operations would formulate a story such

as: “As director of operations, I need to measure my resource utilization by month and by business

practice in order to ensure that I maximize my revenues”.

This simple story already allows us to determine the various roles, subject area, metrics, and

dimensions. In addition, such a story allows the project team to ask business related questions allowing

small increments of knowledge to be transferred to the project team and better understand the

business dynamics.

Subsequent questions would possibly be to explain what is meant by measure, how is resource

utilization defined, what are the business practices, etc. The objective is not to drill deep into each

discussion this early in the process but simply to note the details behind the stories so they can then be

prioritized. It is critical to remember that the level of details will increase as we progress through the

various stages of the project.

During the definition phase, the project team meet with the business representative to determine and

agree on the priority of the team for the duration of the release. It is critical to note that the intent is not

to agree on the details during this stage but the high level direction the team will follow for the duration

of the release. For example, during a recent project done for software development organization the

general manager who was acting as the business representative decided to focus primarily on KPIs for

his consulting practice instead of his product development and training departments. He felt that the

competitive pressures were stronger in his consulting unit and as such, the information he would obtain

from his BI application would be more valuable in the current context. It is critical to remember that we

are not trying to segment the development process by business unit but simply to help with the

definition of the objectives and requirements of each business unit.

6 User Stories Applied: For Agile Software Development by Cohn, 2004



Plan Planning consists of selecting the high level stories to be detailed further from the product backlog. The

objective is to select enough of the highest value stories to be investigated further within the current

project.

At the beginning of the sprint, the team selects items from the backlog. The items with the highest value

are typically considered highest priority. The objective is to work on the highest yielding story while

showing results as quickly as possible. It is critical to focus on the content of the current sprint and not

to spend effort on planning and defining further that current sprint as business priorities might change

the value.

During the planning phase, the development team comes together to assess the efforts associated with

the selected stories. The objective of this phase is to estimate the efforts associated with each story in a

relative measure.

Once the stories are defined, the product owner needs to quantify the value of the cards. The measure

is not as important as the outcome. Some organizations use points, play money or even poker chips to

quantify the value of the stories. Although the process of quantifying can be fun, we are mostly

interested in the outcome. The stories are then sorted in descending value.

As such, stories such as calculating the employee satisfaction rate is certainly more complex than

reporting on the company’s sales, especially when the team suspect that the underlying information

may not be available or accurate.

In addition to needing to define and express his own needs and requirements, the product owner has to

have the ability to prioritize his stories appropriately in order to help the development team achieve

success. Since the AGILE approach is strongly tied to delivering business value, it is important that the

business owner be able to properly understand and communicate the value of each of the stories.

Contrary to a traditional approach where the value of a BI project is estimated at the beginning when

funding is established but not revisited unless additional funds are required, the AGILE approach brings

up the value discussion throughout the development cycle. In addition to helping to determine the

priority of the various stories, establishing value for each story helps to ensure alignment across the

business while avoiding building components that are not necessary. During the planning of each sprint,

asking the question “why are we doing this?” will help bring to the surface the real requirements and

keep in mind the importance of the team goal.

In addition, instead of simply tracking ongoing cost of the project, keeping track of the value of the

components delivered allows the project team to measure the cost and benefits at any point in time of

the project. As such, instead of spending money to develop requirements that may have been

determined months or years before, the question of needing to invest in additional metrics helps the



organization ensure they spend wisely. Instead of the approach of determining the scope upfront and

defining the budget and timelines as a consequence, we define the budget first and play with the scope.

Hence, why the benefits of each story are so critical.

In addition to spending too much time on planning, analysis, modeling and architecture, the proposed

approach reduces this overhead by using a non-superfluous approach where the project team focuses

from beginning to end on the selected priority. Where the traditional approach recommends defining

the entire scope of the project upfront and spends up to 70%7 of the efforts on the Extract-Transform-

Load (ETL) before the business users actually see results, an AGILE BI approach focuses on delivering

incremental value.

Build The building phase takes the project team to a more granular level of details and requires the team to

move to the next level of the project.

Much happens during the building phase where each member on the development team selects stories

on which he/she will work for the duration of the sprint. Typically stories can be divided along two lines

where the developers take responsibility for development in addition to architecture and modeling

tasks.

Another requirement for a successful implementation is to ensure continuous testing and integration of

the components built. It is recognized that data quality is a critical success factor for a BI project and as

such, it remains critical to ensure that additional components built do not break existing scripts and

negatively impact data quality. As opposed to testing at the end of the various releases which is often

the case in traditional projects, continuous integration and testing ensure that testing occurs every time

a new component needs to be integrated.

In a traditional project, much effort is spent fixing quality of the underlying data even if only 10% or 20%

of all the data will actually be used. The incremental approach deals with inaccurate data just when it is

required. As such, the development team spends time working on fixing issues only when it is required.

Timely design and architecture prevents the development team from over architecting the solution and

relies on a non-superfluous solution. Perfection is difficult to reach and the cost for trying to achieve it

hardly justify spending it.

It is expected that the team will encounter some un-anticipated issues during the project. As such, it is

always possible to integrate some of the fixes required in the next sprint planning session. Similarly

performance improvements or database refactoring will be integrated within the various sprints.

To ensure success you have to make sure refactoring is a key component of your approach as you will be

tweaking and tuning your solution throughout multiple sprints and since it is better to present

7 The Data Warehouse Lifecycle Toolkit, Kimball & all., 2008



something to the business users than nothing at all, the development team needs to factor in some time

to fix some of the underlying issues all of this while delivering on an ongoing basis.

Present and Deploy Everyone will agree that the real test is when actual users get to use the system. It is therefore expected

that at the end of each sprint, the development team presents the various metrics that have been

developed during the sprint to the business users. This practice has a few clear benefits in that unless

metrics can be demonstrated in the application – not on paper or in theory – and shown to provide the

expected information, they are not completed. As opposed to asking developers if their work is done, it

is much more accurate and telling to see the performance indicators and determine if they do meet the

expectation or not.

In addition, having to demonstrate the Key Performance Indicators (KPIs) within the application requires

that the proper testing has taken place otherwise the development team will have to present a faulty

indicator or dashboard.

Finally, being able to present indicators early in the process helps the end user better understand the

development team’s capabilities and fuels better business discussion which will eventually lead to

additional stories being added to the backlog.

Many BI initiatives have been cancelled or lost much credibility once decision makers realized that the

data presented was inaccurate. Unfortunately these situations come to the surface late in the game in a

waterfall approach. When using an iterative approach, inaccurate data quickly come to the surface and

are addressed quickly for the project to continue.

To ensure the development team is on the right track, they really need working software in front of

business users at each stage. Once end users see how things work, they quickly move to a deeper

understanding of the process and are in a better position to properly define their expectations. Delivery

of completed KPIs gives business users the opportunity to see and experiment with the KPIs which then

allows them to adjust their expectations and eventually their requirements.

With the traditional approach a substantial amount of time is spent at the end of the project to test and

integrate all components. As in other development processes, the value of identifying and resolving

issues early in the process is always less expensive and more efficient as it usually highlights weaknesses

in knowledge or skills and are addressed sooner to avoid repetition throughout the project.

Finally, the transition to the production environment is usually very intense and as such reducing the risk

by releasing smaller components more frequently not only ensure a smoother transition but also helps

develop the right processes to increase efficiency.



Retrospect The final step of the iteration is the retrospection which has a few objectives where the most important

one is to allow the team to reflect on the successes and determines which areas need to be improved

prior to entering the next sprint. As such, the team collectively assesses its own performance and

determine the best way to adapt in order to successfully achieve its next sprint.

The approach is also very useful from a learning and growth perspective where many organizations

undertake BI projects without having all the necessary experience and skills. Short and frequent

development cycles allow the development to build the required skills, learn from the process and

improve their technical and business expertise.

Conclusion The AGILE approach heavily relies on business involvement within the project team and throughout the

duration of the BI project. Having direct contact with the business team, the development team will

more easily understand the expectations and will increase its knowledge of the business which will make

a more knowledgeable and productive team altogether.

In order to achieve our objectives to: reduce the initial investment, accelerate the availability of the

information, deliver the right solution to the business users and constantly deliver value to the

organization we explained that an AGILE approach is the right solution. Through iterations and

increments, we use a “slicing approach” to develop from source systems to presentation layer in order

to deliver on the business users’ requirements.

The benefits of an AGILE approach to a BI project are substantial. From our perspective, the most

important one for the organization is that it forces the project team to focus on ROI and deliver high

value by spending efforts on activities that will bring the highest value.

By reducing the duration of the development cycles the business and development team can modify

their release strategy and focus on new priorities that may arise as a consequence of a changing

business environment.

Documents

Improving ROI and Success Rate of Your Business Intelligence Project