Business Intellegence Journal

8/11/2019 Business Intellegence Journal

1/60

EXCLUSIVELY FOR

TDWI PREMIUM MEMBERS

volume 19 number 1

THE LEADING PUBLICATION FOR BUSINESS INTELLIGENCE AND DATA WAREHOUSING PROFESSIONALS

Hw bI maks Fd f th Pm efcit ad effcti 4Hugh J. Watson

Fi Gidig Picips fraiig th Pis f big Data 8Bhargav Mantha

bI bst Pactics: Thghy Thik It Thgh 12Max T. Russell

Agi bsiss Itigc:A Pactica Appach 15Justin Lovell

Wats ad Sii: Th ris fth bI Sat machi 23Troy Hiltbrand

Data Gac Gaicati 30Justin Hay

bI Cas Stdy: Stafd Gadat Schf bsiss bids opatia bI f Datai th Cd 36Linda L. Briggs

Q&A: byd Aaytics ad big Data i bI 39

rspsi Dsig: Th Ky trspsi mi bI Appicatis 42Markus Guertler

bI epts Pspcti: mi bI 50Jake Freivald, Suzanne Hoffman, Cindi Howson,and Nic Smith


2/60

BI Training Solutions:As Close as Your

Conference Room

tdwi.org/onsite

TDWIONSITE EDUCATION

TDWI Onsite Education brings our vendor-neutral BI and DW training to companiesworldwide, tailored to meet the specific needs of your organization. From fundamental

courses to advanced techniques, plus prep courses and exams for the Certified BusinessIntelligence Professional (CBIP) designationwe can bring the training you need directly

to your team in your own conference room.

YOUR TEAM, OUR INSTRUCTORS, YOUR LOCATION.

Contact Yvonne Baho at 978.582.7105

or [email protected] for more information.


3/60

1BUSINESS INTELLIGENCEJournal vol. 19, no. 1

volume 19 number 1

3 F th edit

4 Hw bI maks Fd f th P m efcit ad effcti

Hugh J. Watson

8 Fi Gidig Picips f raiig th Pis f big DataBhargav Mantha

12 bI bst Pactics: Thghy Thik It ThghMax T. Russell

15 Agi bsiss Itigc: A Pactica AppachJustin Lovell

22 Istctis f Aths

23 Wats ad Sii: Th ris f th bI Sat machiTroy Hiltbrand

30 Data Gac GaicatiJustin Hay

36 bI Cas Stdy: Stafd Gadat Sch f bsiss bids opatiabI f Data i th CdLinda L. Briggs

39 Q&A: byd Aaytics ad big Data i bI

42 rspsi Dsig: Th Ky t rspsi mi bI Appicatis

Markus Guertler

50 bI epts Pspcti: mi bIJake Freivald, Suzanne Hoffman, Cindi Howson, and Nic Smith

56 bI StatShts


4/60

2 BUSINESS INTELLIGENCE Journal vol. 19, no. 1

volume 19 number 1

eDITorIAl boArD

editia DictJames E. Powell, TDWI

maagig editJennifer Agee, TDWI

Si editHugh J. Watson, TDWI Fellow, University of Georgia

Dict, TDWI rsachPhilip Russom, TDWI

Dict, TDWI rsach

David Stodder, TDWI

Dict, TDWI rsachFern Halper, TDWI

Assciat edits

Barry Devlin, 9sight Consulting

Mark Frolick, Xavier University

Troy Hiltbrand, Idaho National Laboratory

Claudia Imhoff, TDWI Fellow, Intelligent Solutions, Inc.

Barbara Haley Wixom, TDWI Fellow, University of Virginia

Adtisig Sas:Scott Geissler, [email protected], 248.658.6365.

list rtas:1105 Media, Inc., offers numerous e-mail, postal, and telemarketing

lists targeting business intelligence and data warehousing professionals, as well

as other high-tech markets. For more information, please contact our list manager,

Merit Direct, at 914.368.1000 or w ww.meritdirect.com.

rpits:For single article reprints (in minimum quantities of 250500),

e-prints, plaques, and posters, contact: PARS International, phone: 212.221.9595,

e-mail: [email protected], www.magreprints.com/QuickQuote.asp.

Copyright 2014 by 1105 Media, Inc. All rights reserved. Reproductions in

whole or in part are prohibited except by written permission. Mail requests to

Permissions Editor, c/o Business Intelligence Journal, 555 S Renton Village

Place, Ste. 700, Renton, WA 98057-3295. The information in this journal has

not undergone any formal testing by 1105 Media, Inc., and is distributed without

any warranty expressed or implied. Implementation or use of any information

contained herein is the readers sole responsibility. While the information has

been reviewed for accuracy, there is no guarantee that the same or similar

results may be achieved in all environments. Technical inaccuracies may resultfrom printing errors, new developments in t he industry, and/or changes or

enhancements to either hardware or software components. Printed in the USA.

[ISSN 1547-2825]

Product and company names mentioned herein may be trademarks and/or

registered trademarks of their respective companies.

President Rich Zbylut

Director, Online Products Melissa Reeve& Marketing

Graphic Designer Michael Boyda

President & Neal VitaleChief Executive Officer

Senior Vice President & Richard VitaleChief Financial Officer

Executive Vice President Michael J. Valenti

Vice President, Finance Christopher M. Coates& Administration

Vice President, Erik A. LindgrenInformation Technology &

Application Developmen t

Vice Presiden t, David F. MyersEvent Operations

Chairman of the Board Jeffrey S. Klein

rachig th StaffStaff may be reached via e-mail, telephone, fax, or mail.

E-mail:To e-mail any member of the staff, please use thefollowing form: [email protected]

Renton office(weekdays, 8:30 a.m.5:00 p.m. PT)Telephone 425.277.9126; Fax 425.687.2842

555 S Renton Village Place, Ste. 700Renton, WA 98057-3295

Corporate office(weekdays, 8:30 a.m.5:30 p.m. PT)Telephone 818.814.5200; Fax 818.734.1522

9201 Oakdale Avenue, Suite 101, Chatsworth, CA 91311

Business Intelligence Journal(atic sissi iqiis)

Jennifer AgeeE-mail: [email protected]/journalsubmissions

TDWI Pi mship(iqiis & chags f addss)

E-mail: [email protected]/PremiumMembership425.226.3053Fax: 425.687.2842

tdwi.org
http://tdwi.org/http://tdwi.org/


5/60


Go big or go home. In this issue of the Business Intelligence Journal, we look at big

and small from a variety of perspectives.

Big data is getting big buzz, and Bhargav Mantha looks at five guiding principles

to help your enterprise make the smartest investment in, and realize the promises

of, big data. Mantha stresses the importance of using the right tools and looking at

current technologies, such as social media.

Senior Editor Hugh J. Watson looks at five lessons learned from a BI project at a

nonprofit and the benefits, including big reductions in delivery time for critical

information.

Troy Hiltbrand foresees big changes ahead in how business intelligence is executedand deployed as smart machines and automation invade areas traditionally unique

to human interaction.

Justin Lovell looks at the big back log of BI projects most enterprises have and

explains how teams can implement the agile mindset when building data output

applications. Lovell explores several agile concepts and how they specifically relate

to business intelligence projects.

Justin Hay writes about the big gamegamification, that is. He proposes an

alternative to the traditional data-governance-by-committee approach by applying

principles of the gamification movement.

We also look at why small is just as important as big. Max T. Russell looks at how

your attention to the smallest detail can prevent big problems. Markus Guertler

discusses the importance of using responsive design for building mobile BI applica-

tions on small form factors. Our Experts Perspective feature discusses best practices

for moving to mobile BI.

Finally, this issues case study describes how the Stanford Graduate School of Busi-

ness realized significant performance improvements to some core operations and

how that success is paving the way for pushing deeper into the cloud. In our Q&A,

we examine Barry Devlins idea of business unintelligence.

As always, we welcome your comments, both big and small. Please send them to

[email protected].

From the Editor
mailto:[email protected]:[email protected]


6/60


How BI MakesFood for the PoorMore Efficient andEffectiveHgh J. Wats

Itdcti

ese are challenging times for nonprofit organizations.

e troubled economy has increased the need to deliverservices efficiently while also making fundraising more

difficult. Its the classic need to do more with less. is

climate has a lso contributed to the expectation that

nonprofits be highly transparent about how their funds

are spent and the benefits realized.

In judging TDWIs Best Practices Awards in the govern-

ment and nonprofit category last year, I was impressed

with the work that Food for the Poor, an international

relief and development organization, is doing, and how

it is using business intelligence (BI) to help the entireorganization work more effectively and efficiently.

BI enables direct marketers to maximize the efficiency

of their appeals and outreach to donors

It gives managers visibility into key operational and

financial information

It provides real-time access to information about

history, goals, achievements, and targets to track

performance

It uses automatic scheduling and alerting technology

to keep the staff apprised of daily donations

By reducing the time it takes to access and deliver critical

information, BI makes the staff more efficient, reduces

the total operating budget, and frees up more time for the

organizations mission. Every minute wasted on manual

FooD For THe Poor

Hgh J. Watsis a Professor of MIS

and holds a C. Herman and Mary Virginia

Terry Chair of Business Administration

in the Terry College of Business at the

University of Georgia. He is senior editor

of the Business Intelligence Journal.

[email protected]


7/60


FooD For THe Poor

reporting is time lost in meeting the needs of the people

Food for the Poor serves.

I invited two of the major contributors, Vickie

Torregrossa (IT director) and Jamil Idun-Ogde (data

analyst/project manager), to describe BI at Food for the

Poor and share what they have learned that might be

helpful for other organizations.

At Fd f th P

Food for the Poor is the largest international relief and

development organization in the U.S. according to the

Chronicle of Philanthropy. Founded in 1982, its inter-

denominational Christian ministry serves the poorest ofthe poor in 17 countries throughout the Caribbean and

Latin America. Its programs provide housing, healthcare,

education, fresh water, emergency relief, and microen-

terprise assistance in addition to feeding hundreds of

thousands of people each day. Since its inception, Food

for the Poor has provided more than $10 billion in aid

and is one of the most efficient nonprofit organizations

in the U.S., with more than 95 percent of all donations

going directly to programs that help the poor.

bI at Fd f th PFood for the Poor has implemented a BI environment

that empowers business users throughout the enterprise

to gather information and gain insight through user-

friendly dashboards and parameterized reports that

allow users to better focus on their missions. Having

real-time access to information on finances, history, goals,

achievements, and appeals has allowed Food for the Poor

to reduce its total operating budget and respond more

effectively to catastrophic events such as the earthquake

that rocked Haiti in 2010 and Hurricane Sandy in

2012. e BI environment has also had a positive impacton direct mail, donor relations, and other fundraising

activities that help the organization respond quickly to

humanitarian emergencies.

BI Environment

Food for the Poors nine-person IT department is

responsible for operational systems and BI. e director,

a data analyst/project manager, and a programmer each

spend part of their time on BI reports, dashboards, and

special analyses.

e nonprofit does not have a data mart or warehouse.

Instead, it uses Information Builders WebFOCUS BI to

access, analyze, and display live data from operational

systems. Data sources include the donor, supply chain,

and departmental systems on a variety of platforms

(e.g., IBM Power Systems), databases (e.g., FoxPro), and

systems and applications (e.g., Microsoft Dynamics NAV

and Excel).

BI Applications

Since acquiring WebFOCUS, the IT department hasrolled out reports and analytic tools throughout the

organization. For example, direct marketing personnel

can track donations and send real-time reports to key

individuals managing the operation and spearheading

campaigns. Comparisons are made between the effective-

ness of this and previous years campaigns. According to

Carlton Lewis, director of direct mail, e reports make

it easier to compare daily income from different appeals

and to monitor the effectiveness of various campaigns,

including the response to different acquisition pieces and

lists. During busy periods, the BI tools help supervisorsbalance the caseload and track donations, exporting data

to Microsoft Excel for analysis as necessary.

Food for the Poors marketing department uses the BI

environment to manage its new TV monthly giving

campaign. e reports are making it easier to track the

effectiveness of the campaign over time.

About three years ago, Food for the Poor implemented a

new ERP system, Microsoft Dynamics NAV. rough

a combination of the BI tools and the new system, usersare able to greatly enhance the tracking of goods received,

who donated them, and where they are being distributed.

One report shows the contributions of different vendors

over time and may reveal, for example, a vendor who

has not donated recently and should be contacted. ey

can see if there are countries that have received more or

less help compared to previous years. What used to be a

tedious, manual process is now automated, freeing staff

to acquire more goods to help the poor.


8/60


It is now possible to track the flow of donations from

specific vendors to specific countries and specific people.

For example, if there is a need to recall a product, theorganization can identify who received the container and

quickly issue the appropriate recall.

e accounting department uses the BI environment

to track daily deposit information, forecast cash-flow

requirements, and create a variety of financial reports.

Formerly I had to write queries to answer questions, and

it was very cumbersome, says Jeff Alexander, Food for

the Poors controller. I now have several parameterized

reports that I can run to quickly produce results.

Alexander has also created parameterized reports for his

staff, making the entire department more efficient. ey

can run the reports on their own and output the results

directly to Excel spreadsheets, he adds. WebFOCUS

helps them get daily deposit information more quickly

and it speeds up the monthly closing process.

BI reports and dashboards also boost efficiency for the

projects department, which works with each country to

oversee specific types of relief projects. ese employees

must continually monitor income by category suchas animal husbandry, housing, food, and medical. A

projects dashboard reveals the type and quantity of goods

received by displaying information about the number,

amount, and types of gifts from each donor. It tracks

progress toward goals for each category.

e donor relations department depends on the BI

environment to maximize the efficiency of essential

fundraising activities. e BI implementation literally

gave me back about seven days of my life each month,

says Michael Chin Quee, director of donor relations.at was the time it would take to glean the information

from the necessary reports to manage my staffs produc-

tivity. Compiling the data was done manually, which was

time-consuming and stressful. I can now accomplish in

one day what took several daysand with the assurance

of accuracy.

Every day the donor relations department receives e-mail

messages about gifts that came in the previous day.

FooD For THe Poor

Especially in the case of large donations, the department

contacts the person or organization and thanks them for

their gift. e ability to make calls quickly after dona-tions are made has increased subsequent donations.

bsiss Ipact

Food for the Poor has achieved tremendous operational

efficiency, with more than 95 percent of all donations

going directly to programs to help the poor. Senior

officers endeavor to maintain this outstanding efficiency

rating while increasing the level of funds collected from

donors. Achieving this objective means leveraging one

of the highest cost centers in their organizationdirect

marketingcost effectively. is was one of the chal-lenges that catalyzed the organization to acquire BI

technology. ey wanted to permit instant visibility

into their marketing database while enabling the staff to

quickly evaluate such activities as direct mail, radio, and

advertising campaigns.

lsss lad

Food for the Poors experiences provide insights that may

help other organizations. Some insights are especially

appropriate for nonprofits.

Lesson #1: The Right Software Purchase Can Save Money

in the Long Term

Like any nonprofit organization, Food for the Poor has

to be particularly careful about how it spends its money.

at said, the organization is a relatively large nonprofit

and wants to invest wisely in business best practices.

ere was a clear business need for the information

that could be provided by BI, and senior management

approved BI investments after the potential benefits were

communicated, which included productivity gains and

increased efficiency.

Lesson #2: Deploy Quickly

It is always good to be able to realize the benefits from

IT investments quickly. Because Information Builders

allowed Food for the Poor to start using its software

during the proof of concept and there was no data mart

or warehouse to build, it was possible to implement

reports and dashboards quickly and to begin reaping the

benefits of BI.


9/60


Lesson #3: Users Love Live Data

Because the dashboards display live data, users can access

information as it happens. For example, it is possible tosee what items are going to be shipped from suppliers

and Food for the Poors warehouses and track the items

movement to the final recipients. Not all data is live.

Historical data for reporting purposes is sourced from

IBM Power Series and Microsoft Dynamics NAV.

Lesson #4: Dont Wait for Requests

Determine the business needs and start building the

system. It can always be modified later. By showing

people whats available, you can increase interest and

excitement, which can ultimately lead to additionalrequests.

Lesson #5: There Is a High Level of Satisfaction in

Working for a Nonprofit

When you work for a nonprofit organization, you

constantly see the need and urgency to provide assistance.

ere is a good feeling associated with going to work.

Food for the Poor has a monthly staff meeting where

employees are informed about the good being done in

the countries served. People know that their efforts are

helping people in need and believe in the work that theyare doing.

Ccsi

Many nonprofit organizations undertake advanced

humanitarian efforts but lack the IT systems needed to

sustain the efforts in a significant way. As an established

nonprofit corporation, Food for the Poor is concerned

with efficiently tracking and reporting the results of its

campaigns and appeals for voluntary donations.

Using BI dashboards and reports, Food for the Pooris improving operational efficiency in nearly every

department, with a positive impact on direct mail, donor

relations, fundraising, accounting, logistics, and project

management. e BI tools help the organization keep its

overhead down and respond more quickly to humanitar-

ian emergencies. Reports that used to take hours to

produce are now run in minutes. e systems ease of use

allows managers to obtain critical operational informa-

tion with little or no assistance.

FooD For THe Poor

Food for the Poors BI environment is helping the

organization to operate more efficiently and improve

fundraising efforts in the wake of catastrophic events. Forexample, in the six days following the earthquake that

virtually destroyed Port-au-Prince, Haiti, the organiza-

tion began feeding hot meals to about 20,000 people per

day and distributing thousands of tons of relief items

such as food, clothing, and medical supplies. e effort

continues to this day, in Haiti and elsewhere, guided by

the current information and insight delivered from Food

for the Poors information systems.


10/60


AlIzInG THe PromISe oF bIG DATA

bhaga mathais a manager at

ZS Associates and a leader of the firms

global business intelligence practice.

[email protected]

Five GuidingPrinciples forRealizing the Promiseof Big Databhaga matha

Astact

mst ganizatins chaactiz ig data in ts f vu,

vcity, and vaity, ut it is usfu t cnsid igdata in th sa way w k at infatin anagnt,

anaytics, and hw thy ipact usinss dcisins. Aft a,

big data is a swping t that incuds a vaity f nt-

pis cncns, f anaging and scuing data sts t

tchngis that can anayz th data quicky and thus

nhanc usinss vau.

In this atic, w utin v guiding pincips t hp

cpanis ak pudnt invstnts and aiz th pis

f ig data. businsss shud us ths guidins t hp

th think had aut whn, wh, and hw t st aiz

ig datas vau within thi ganizatins.

Itdcti

Hardly a day goes by without some mention of big data

in our lives. e hype-versus-hope debate of big data will

continue for some time as organizations across industries

grapple with the questions of why big data is important,

what to do with it, and how to get started.

Although big data is most easily characterized in terms

of high volume, velocity, and variety, it is more practicalto define big data by the way we think about information

management and analytics and how they impact business

decisions.

One of the biggest obstacles organizations face is think-

ing big data is an initiative when, in fact, big data is an

umbrella term that covers many problem spaces, data

sets, technologies, and opportunities for enhancing

business value.


11/60


reAlIzInG THe PromISe oF bIG DATA

Here are five guiding principles to help your enterprise

avoid becoming overwhelmed by the hype and focus

instead on making prudent investments that will help yourealize the promise of big data.

Picip #1: Dti th siss cas st.

A critical step for key executives to ensure big data

adoption is to identify the business initiative and quantify

tangible business value. is involves pinpointing which

parts of the business would benefit from expanding

available data to provide more complete answers. For

example, a brand manager investigating a decline in

sales may want to augment the analysis by integrating

insight from call center records, Web logs, and consumersentiment through social media commentary on quality,

functionality, or price.

Key executives may also determine if big data analytics

can help monetize a portion of their business. For

example, they may use analytics to immediately make a

relevant offer after a credit card is used to initiate another

transaction instead of storing the transaction for later

reporting.

e business cases for investing in big data vary. eycan be business-process-specific, such as improving the

customer experience, optimizing R&D, or managing

IT. ey can be industry-specific, such as optimizing

price or channels for technology firms, detecting fraud

for the financial services industry, managing intellectual

property for media companies, or improving treatment

outcomes for healthcare providers.

Organizations often take a misstep by thinking that big

data is just another source for business intelligence (BI).

For example, one organization confessed to using theirbig data pilot to build Facebook and Twitter interfaces to

gather social media data, but said the effort was unsuc-

cessful because executives failed to consider what to do

with that data. ey didnt determine at the outset how

to process the data, what questions it could answer, and

what analytics were required to make sense of it (senti-

ment analysis, monitoring evolving topics, or uncovering

networks and relationships).

Finding a business-driven initiative with measurable out-

comeswhether improved customer retention, increased

revenue from improved sales/channel productivity, oreven cost reductionwill improve your organizations

success rate with big data initiatives.

A critical step for key executives

to ensure big data adoption is to

identify the business initiative and

quantify tangible business value.

Picip #2: us th ight ts ad tchgis.

Organizations should consider four main capabilities

to expand their existing BI and analytics initiatives to

support big data analytics.

e most important capability is advanced analyticsto

uncover previously hidden patterns. With new types of

data comes the need to apply new types of algorithms,

such as entity analytics, network analytics, text analytics,and real-time scoring. Scalability is important because

improved accuracy and trust in your data means your

users are more likely to want to integrate additional data

sources or increase data volumes. Analytics must be

able to push these algorithm processes to interpret text,

images, and video streams.

Visualization and explorationcan help your enterprise

find more complete answers to business questions. New

types of data (and greater volume) increases the need for

new forms of visualization (such as heat maps) to presentthe data to users and highlight important patterns. Tools

such as Tableau Software and Datameer enable interac-

tive, iterative, search-like, visual data discovery.

e third capability is to turn insight into action

to drive a decisioneither with a manual step or an

automated process. Applying analytics to streaming big

data requires technology that uses predictive models and


12/60


AlIzInG THe PromISe oF bIG DATA

business rules to automate decisions and identify outliers

where business judgment is needed.

Finally, analytics tools must assemble the right mix of

informationin a way that makes sense to the business.

is may include:

Tooling to compose fast-performing queries on very

large data sets or to access high-performance analytic

databases such as Aster Data, EMC Greenplum, or

IBM Netezza

Analytic processing capabilities to ingest data in

motion, apply filters, and surface relevant real-time data

Query and process returned data from unstructured

data (for example, in HDFS, the Hadoop Distributed

File System)

Big data requires more than just Hadoop. Although that

open source software framework has the greatest name

recognition, big data is too varied and complex for a

one-size-fits-all solution. Other classes of technologies

are equally well suited to managing big data, such asNoSQL (not only SQL) and MPP (massively parallel

processing) stores.

Again, what matters is which of the three Vs poses the

greatest challenge for you and which of these technologies

supports the business case. In fact, there is no require-

ment for you to invest in your own infrastructure.

Instead, you might explore options for a cloud-based

service, such as Google BigQuery, and save on infrastruc-

ture costs.

Picip #3: Pp, pp, pp.

After youve developed the business case for big data,

begin a thorough skills assessment, because newer

analysis techniques and technologies may require differ-

ent skills or talent. ere are three particular roles (and

associated competency models) that you can define for a

big data initiative:

e data scientist, who applies his or her statistical,

mathematical, and computer science skills to work on

large, complex data sets to find, interpret, and distrib-ute statistically significant information. He or she will

also ensure that significance is easily understood and

acted upon by others.

e business analyst, who blends business

understanding with data acumen to determine what

information is important for the business and how to

bridge the IT or data science gap.

e technologist, who has the skills needed to

identify and assemble the best set of big data technolo-gies and developers (for example, Hadoop and Hive)

to deliver on the business initiative.

Notice that the skills required dont all need to be about

Hadoop and advanced algorithms. One of our clients

admitted to feeling overwhelmed by the hype, leading

them to think of big data initiatives as beyond the

companys technology skills. In fact, a ll the client was

looking to do was gain insights from clickstream data,

which did not require Hadoop or the ski lls of a data

scientist. Mapping the business case, determining thetechnology needed, and obtaining the appropriate skill

sets helped the client overcome their fear and make the

right investments toward big data analytics.

Picip #4: Stat thikig scia.

Big data could be an important component of your social

media strategy, especially when it comes to understanding

customers, prospects, and key influencers. Social media

allows for ongoing engagement that can provide near-

real-time insight into customer attitudes and behavior.

Analysis of socia l media data can help you rapidlyidentify trends: who uses your solutions, what customers

and prospects think about your and your competitors

brands and solutions, and what emerging markets are

developing.

Recognizing the value of and leveraging social media

data sources are relatively new challenges for many

organizations. A social intelligence effort wil l require

rethinking and redesigning existing information manage-


13/60


reAlIzInG THe PromISe oF bIG DATA

ment ecosystems. New analytical platforms, techniques,

tools, and governance processes are needed to unlock

customer insights.

e implementation of a socially enabled business

through big data includes three main steps:

1. Listeningto consumer dialogue on social networks,

sites, and communities and collecting the data

2. Analyzingthe gathered information (mostly opinions)

and applying natural language processing algorithms

to extract actionable meaning and the most recurrent

themes

3. Engagingwith customers by closing the loop and

taking quick, decisive, and appropriate actions based

on gathered insights

Picip #5: Dt tat ig data as issi citica

ight away.

Although big data quality will become increasingly

important, dont treat social media data or wiki data

like mission-critical financial data right away. Apply the

appropriate level of control to its use and exposure.

Your initiative may well be stifled from day one if you

apply the rigor of initiating and managing traditional

data warehouse projects. Instead, help the process be

iterative and collaborative: let the business and IT explore

interesting sources of data, refine what is important, and

apply the appropriate algorithms. Better outcomes are

possible when an organization conscientiously allows

big data initiatives to be iterative, exploratory, and even

transient in some cases.

Th big Data opptity

Big data presents a growing opportunity to understand

and change interactions with customers. It allows

companies to improve existing business processes, to

launch new lines of business, and to reevaluate how and

why data can improve decision-making processes. Using

these guidelines, think hard about when, where, and how

to best realize big datas value within your organization.


14/60


THInKInG IT THrouGH

ma T. rssis the owner of Max and

Max Communications. He works behind

the scenes to promote individuals and

projects in a variety of industries.

[email protected]

BI Best Practices:oroughly inkIt roughma T. rss

Avid an aush f yu bI pjct y thinking thugh vy

iaginaand uniaginadtai.

BI expert Alexis was hired by a nationwide adoption

agency to build a dashboard for user management. eIT director told her during the interview that if this first

project went well, the CIO would approve a cautious BI

expansion throughout the organizationunder Alexis

leadership.

e IT director listened to Alexis BI philosophy,

approved her methodical approach to the dashboard

design, and then wished her well. Several days into the

job, Alexis hit a brick wall. e CIO, who knew just

enough about data architecture to make himself danger-

ous, stubbornly disagreed with her approach.

e project was destined to be a headache to the end

because Alexis had failed to anticipate one variablethat

someone might disagree with her architecture.

A successful BI plan depends on doing many things

right, but certain unanticipated details can cause painful

interruptions or even kill a project. Its worth your teams

time to develop a vivid imagination to discover surprises

that could ambush your plan.

naigat i Y mid

You dont want executives or users to see you floundering

because of a detail you didnt expect or know about.

ats why you and your team must imagine your way

through every conceivable and inconceivable detail of

show-stopping significance.


15/60


THInKInG IT THrouGH

Consider this non-BI example of an environment in

which various details could make or break your efforts

the crawl space under my house. Its very difficult tonavigate. More than once I have crawled through the

fog of spiderwebs while squeezing past one tight, muddy

space after another. A building contractor said its the

worst crawl space hes ever worked in.

ats why I do each repair in my mindbefore I go under

the house. I do notwant to have to start over.

Fixing a leaking water pipe may require:

Pliers, a drill and drill bits, a hammer, a pry bar, aflashlight and trouble light, screwdrivers, nails and

screws, a sled for carrying equipment, wire to sup-

port the pipe, wire cutters, a hat to keep spiderwebs

off my head, extension cords to allow me to crawl

as far as possible without getting lost, a face mask,

a cloth to clean my hands, a foam pad to lie on, a

plastic bottle to support my head while lying on my

back, gloves, and safety glasses to keep particles out

of my eyes.

If I fail to anticipate even one procedure or forget a tool,I might have to make the miserable journey back to the

crawl space opening, pull myself out, find the right tool,

return to the opening, and crawl back to the trouble spot.

I dont always have the heart to go back.

Its worth your teams time to

develop a vivid imagination to

discover surprises that couldambush your plan.

Aticipatig Pss f Ic

Now lets return to the problem Alexis faced with the

dashboard. Neither she nor the IT director had involved

the adoption agencys CIO in discussions about their

approach to the dashboards design. Alexis finally decided

to do it the CIOs way rather than wear herself out in

repeated arguments with him. By then, she had lost

trust with the IT director, who felt he had no choice butto deliver what the CIO would eventually demanda

dashboard that would never do what the adoption agency

needed. e blame would fall on the consultant, Alexis.

One question would have spared Alexis months of

anguish: Who else will be involved in deciding how

this project will be done? e result of not asking the

question was, in her words, a BI failure.

at question would have changed everything by giving

her a chance to schedule a meeting with the CIO (andany other decision makers) to set realistic expectations. If

the CIO still insisted on a faulty dashboard, Alexis could

bow out of the job. She had no interest in doing things

the wrong way!

Furthermore, straight talk prior to being hired might

have been more persuasive, building the CIOs confidence

in Alexiss skill as a true expert who would not stand

silently by and let the agency waste money.

Aticipatig th rtSometimes details go unseen because they seem too tiny

to worry about. Ive found myself in uncomfortable situa-

tions when I incorrectly assumed that an electrical outlet

was within reach of the power cord on my presentation

equipment, or that a projection screen would be available.

ese are easy mistakes to make. ey are also easy to

avoid if you are wil ling to think through the details of

your plan.

Imagine that youre at a BI meeting when the entire

team agrees that the first step of the planning phase is toconnect three department leaders computers so they can

monitor and discuss the same set of planning data.

You ask, Who is in charge of providing the router to

make that happen?

It sounds like a petty question to your teammatesuntil

you explain why youre asking. A certain employee in the

central office moves as slowly as she possibly can when-


16/60


THInKInG IT THrouGH

ever anything of importance depends on her approval.

Pulling a router out of the locked cabinet and assigning it

to the BI team is simple enough, but her modus operandiis to control others by moving at a pace thats just slow

enough to frustrate them and remind them that they

need her.

Because you asked the right question and anticipated

a problem, a BI team member can notify the central

office manager that a router is needed ASAP, preventing

a miniscule variable from delaying your initiative for

a ridiculous 72 hours, as has happened to others. Your

leadership has preserved the projects momentum and the

teams enthusiasm.

Project success means paying attention toand imagin-

ingall conceivable and inconceivable details ahead of

time, no matter how trivial they may seem. Expect the

unexpected and prepare accordingly.

Aticipatig a bach i Ptc

You cant think of every significant stumbling stone by

yourself, of course. A good reporter has a contact list of

anonymous sources to draw on. A good detective has

developed a set of confidential informants. You mustassemble the same support for your project.

A nurse supervisor on the orthopedic floor at a hospital

blew the whistle on a BI tool when she noticed that IT

had given too much access to patient information. e

radiology department read and misunderstood sensitive

doctors notes about a patient, concluded that he would

be nothing but trouble for the hospital, and then declined

services to him.

Not only were the notes supposed to be unavailable tothe radiology department, but orthopedic personnel were

the only ones who could properly interpret them. e BI

tool that was supposed to be a business solution became

a potential loss of revenue, to say nothing of a privacy

violation.

Imagine if that BI tool had been your responsibility.

Imagine if you had developed contacts you could consult

with, so that you were able to bounce ideas off the nurse

supervisor ahead of time. You would have been able to

present her with what-if scenarios and ask what impact

your project would have on the floors operation. You verypossibly could have avoided the breach in protocol. At the

very least, you would have built rapport with the supervi-

sor and others whom you could add to your list of trusted

informantspeople who can help keep an eye on the

effectiveness of your business solutions.

nthig bats a Gat Stat

e beginning moments of a BI plan are where so much

goes wrong or right. Use your imagination to navigate the

plan before presenting or implementing it.

Perform cooperative detective work to discover every

possible obstruction of importance, what other people

know that you need to know, and how to enlist their

support before you begin a clumsy invasion. Nothing

beats a great start.


17/60


18/60


AGIle bI

Dr. Royce recommended against the phase based

approach in which developers first gather all of

a projects requirements, then complete all ofits architecture and design, then write all of the

code, and so on. Royce specifically objected to

this approach due to the lack of communication

between the specialized groups that complete each

phase of work.

e agile methodology and the assembly line do share

one characteristic: they are based on small units of work.

I remember that when I started my career as a Delphi

developer I came across a term that intimidated me

primarily because of its spelling and seeming complexity:polymorphism1, which is the ability of objects belonging

to different types to respond to method, field, or property

calls of the same name, each one according to an appro-

priate type-specific behaviour. After resisting the urge

to rush into coding my phase of a project, I had to take

a step back, break down the requirements into smaller

components, and (after understanding the relationships)

write my code in such a way that it only did one thing.

However, when integrated together with other code, it

met the overall requirements.

In addition to getting us into the object-oriented mindset

of code reuse, this approach dramatically improves the

quality and velocity of subsequent tasks simply because

the less code we write, the fewer bugs we can expect.

Herein lies the secret of the agile approach: breaking

down requirements into small, inter-related tasks that

can be executed quickly, with a high degree of quality,

and completed by different people, regardless of the type

of application were buildingwhether a data entry

application or an output application (one that generatesan analytical report, for example).

In the terminology of the agile methodology, breaking

down requirements into smaller tasks is cal led creating

stories. To illustrate, consider J.R.R. Tolkiens e Lord

of the Rings. is epic consists of many stories connected

by various characters and situations that are all woven in

and around the overall plot. All of these stories merge to

create an imaginary world that was years in the making.

In a similar sense, the stories we support in our various BI

functions are tied to overall epics within the business that

need to be delivered to achieve success.

bakig rqits it Stis

When a business user comes to the BI team with a

requirement, it is the teams responsibility to do two

things: (1) validate the requirement and (2) break the

requirement down into stories.

If we as BI teams are to add intelligence to a business,

then we must validate user requirements up front. It is

not necessary to provide an extensive explanation of how

to conduct business analysis. Instead, simply ask therequestor to complete the following three statements:

As a ... (role) ... I want ... (thing) ... In order to ... (purpose) ...

If the requestor cannot provide clarifying details to these

simple statements, then the development team needs to

push back and decline the request until these statements

can be completed. If clarity isprovided, then the team

can determine if the request has already been completed,and if not, determine the value it offers to the business

(which in turn affects its priority).

Herein lies the secret of the

agile approach: breaking down

requirements into small, inter-

related tasks that can be executedquickly, with a high degree of

quality, and completed by different

people, regardless of the type of

application were building.

1 See http://www.princeton.edu/~achaney/tmve/wiki100k/docs/Polymorphism_in_object-oriented_programming.html
http://www.princeton.edu/~achaney/tmve/wiki100k/docs/Polymorphism_in_object-oriented_programming.htmlhttp://www.princeton.edu/~achaney/tmve/wiki100k/docs/Polymorphism_in_object-oriented_programming.htmlhttp://www.princeton.edu/~achaney/tmve/wiki100k/docs/Polymorphism_in_object-oriented_programming.htmlhttp://www.princeton.edu/~achaney/tmve/wiki100k/docs/Polymorphism_in_object-oriented_programming.html


19/60


AGIle bI

When the request rises to the top of the priority list, then

together the team breaks the requirement down into epics

and their stories. In an effort to put epics and stories intocontext, think of an epic as a package of work and a story

as an individual use case (seehttp://en.wikipedia.org/

wiki/Use_case). A story can be further broken down into

tasks and subtasks that can be taken on by one or many

team members. is leads us to another property of the

agile methodology: the entire team commits to complet-

ing these defined stories within a defined amount of time,

typically one to two weeks, called a sprint.

e team defines its capacity by assigning points to the

selected stories. For example, a complex story mighttake 20 points, while a simple story might take just 1

or 2 points. Over time, a team will be able to fine-tune

its delivery capacity by reflecting on how many points

to assign to different stories. In my experience, a well-

functioning scrum team of about four people can manage

about 140 points in a two-week sprint.

When it comes to requirements

gathering, breaking requirementsdown into stories, and designing

tasks, the entire team is involved.

Following this approach, agile BI teams can deliveron

weekly or biweekly cyclesstories to the business that

may be part of a larger epic but can still be deployed into

production and shown to the business as having been

completed. Later in this article we will review a BI projectscenario to see how this all works.

Th Ta

You will have noticed that throughout our discussion we

have referred to the team. When it comes to require-

ments gathering, breaking requirements down into

stories, and designing tasks, the entire team is involved.

Enterprises are often conflicted; they want to add more

human resources to a development project to solve the

development backlog but worry that larger teams areless efficient. We might assume a project will take longer

when we must explain its scope and requirements to

more people, but I have found that when team members

understand the requirements and contribute to the design

and planning, actual development, testing, and resulting

quality prove the value of the agile methodology.

Picking up on some key phrases from Dr. Winston

Royces definition, we can unpack the underlying message

regarding how an agile approach can benefit a team.

Sequential development, or in todays project manage-ment terminology, a waterfall approach, provides a

mechanism for clearly outlining the dependencies in

completing a product but endangers the final delivery

of the solution by fostering silos of specialties within the

project team. e end result is that individual specialties

determine the velocity and quality of the solution, and

the entire team does not share the responsibility of the

overall delivery, but instead members are concerned only

with the work assigned to them.

At its core, the agile methodology seeks to unite theproject team (developers, users, and sponsors) in under-

standing requirements and to reduce dependency on

specialized skills by lif ting up the overall team skills and

by fostering communication with a sense of community

responsibility.

How this works in practice is that (ideally) any team

member would be able to tackle a task. is is possible

because the entire team has committed to the delivery

of stories within a sprint. A natural spreading of the

workload occurs; if one team member finishes a task, heor she can either take the next task on the list or assist

fellow team members to complete their tasks.

Can this work in a BI environment? To a large extent,

yes, because if someone is defined as a BI developer, that

implies they understand the data warehouse life cycle and

have been exposed to the organizations BI tool set. Even
http://en.wikipedia.org/wiki/Use_casehttp://en.wikipedia.org/wiki/Use_casehttp://en.wikipedia.org/wiki/Use_casehttp://en.wikipedia.org/wiki/Use_case


20/60


AGIle bI

todays business analysts tend to be far more technical

than were traditional business analysts, which enables

them to provide valuable support in the design, testing,and user acceptance sign-off phases.

Most important, this approach includes the original busi-

ness user who requested the deliverable as a member of

the team. All too often business users are happy to make

demands but less inclined to get their hands dirty in the

delivery. Using different feedback mechanisms (explained

later in this article) will assist the team in regularly

keeping in touch with and showing business users what

work is active and what is complete.

An agile approach can foster team spirit. A team

committed to working in a sprint will have a sense of

accomplishment in seeing work completed and imple-

mented. is can be a motivating force in and of itself.

Th ecti

A word of caution is in order. Do not get lost in lists

of agile best practices. Agile methodology is full of

processes, procedures, and consultants insisting on strict

adherence. My advice to any BI team looking to adopt

an agile approach is to understand the methodologyfirst and then relate it back to what will work best for

your environment in practical terms. Never lose sight of

the objective for adopting agile in the first place, which

should be to manage a happy, lean but mean team whose

members deliver projects effectively, on time, and within

budget. e suggestions that follow have come out of

practical, real-world experience delivering projects using

an agile methodology in a BI environment. ey are not

the suggestions of a die-hard agile guru.

One of the exciting things about using an agile approachis the teams ability to time box deliverables into a

sprinta short period of work, typically one to two

weeks. As has been mentioned, moving toward an agile

approach is not all about an iterative approach but rather

about managing execution via team dynamics. e

iterative-ness of the agile approach comes into play

because to effectively execute with agile, the following

concepts must be adopted and repeated.

Ive never had good experiences

with so-called project managers,

who typically are really just project

administrators who take down the

minutes and set up meetings.

Here is an overview of how agile teams Ive worked with

have been organized as well as a description of the roles ofkey players and the essential meetings held.

Product owner.In our BI environment, we must have a

team leader. In the software development world, the

product owner is a business user or manager. However,

when it comes to BI, there should be only one product

delivered to the business and that is intelligencethat

is, a decision support system.

Scrum master.Ive never had good experiences with

so-called project managers, who typically are reallyjust project administrators who take down the minutes

and set up meetings. e great thing about agile is that

because the team is heavily involved in so many different

steps of the project life cycle, the project in some ways

manages itself. We will, however, need to coordinate

the sessions and someone should chair these and keep

discussions focused. Enter the scrum master. is

person can be someone from within the team or it can

be a role performed on a rotational basis, but it is not

always practical to load a team member with additional

responsibilities when they need to focus on development.is can be even more overwhelming when there are

multiple scrums working on multiple sprints. What works

best in our environment is that the BI team leader fulfill

this role.

Daily stand-ups2.ese 15-minute, early-morning sessions

set up and chaired by the scrum master are essential for

keeping track of progress and impediments. e daily

scrum meeting is not used to solve problems or resolve

2For more information on daily stand-ups, grooming and planning sessions, andreview and retro sessions, see http://www.mountaingoatsoftware.com/agile/scrum/.


21/60


AGIle bI

issues. Issues are taken offline and usually dealt with by

the relevant subgroup immediately after the meeting.

During the daily scrum, each team member answers thefollowing three questions:

1. What did you do yesterday?

2. What will you do today?

3. Are there any impediments in your way?

Grooming and planning sessions.During this 90-minute

session scheduled every two weeks (depending on the

sprint duration), the product owner meets with the teamto discuss stories in the product backlog. e product

owner shares the current known priorities and may ask

the scrum core development team for help in determining

the relative cost and risk associated with any new items or

items for which new information has come in. e scrum

team is also asked to give input on the sequence of the

work and is encouraged to suggest ways to optimize the

order in which work is done.

Review and retro sessions.A 90-minute session is scheduled

every two weeks (depending on the sprint duration).Each sprint is required to deliver a potentially shippable

product increment, so at the end of each sprint, a sprint

review meeting is held. During this meeting the scrum

team shows what they accomplished, typically in the

form of a demo of the new features. No matter how

good a scrum team is, there is always an opportunity to

improve. Although a good scrum team will be constantly

looking for such opportunities, the team should set

aside a brief, dedicated period at the end of each sprint

to deliberately reflect on how they are doing and to find

ways to improve. is occurs during the sprint retrospec-tive. Each team member is asked to identify specific

things the team should:

Start doing Stop doing Continue doing

During a brainstorming session, the team develops an

initial list of ideas and typically votes on specific items to

focus on during the next sprint. At the end of the sprint,

the next retrospective is often begun by reviewing the

list of things selected for attention in the prior sprintretrospective.

Finally, if there is anything that you should remember

from this article, it is the final yet most important

concept in achieving successful story delivery:

Story swapping.e danger in every project is scope

creepchanges in scope midway through a project.

Breaking down user requirements into relevant stories

and committing to the associated story points during a

sprint provides the team a powerful mechanism of controlover the scope. If and when (it is inevitable in every

project) the scope must change, the team need not panic

because it has committed to delivering a certain number

of story points during a sprint. When the requirements

change and new stories need to be defined and allocated,

the product owner enters into negotiation with the

business.

e negotiation will go something like this: e

following new story of eight points has been requested

to be delivered during the current sprint. e capacity ofthe current sprint is full and therefore it is not possible

to add more work during this phase, except on one

condition. e current sprint contains two other stories,

also of eight-point value, that have not been started. If the

business would like to prioritize and select which of these

two stories is to be swapped out with the new story, the

team will still be able to meet the delivery deadline.

In this scenario, the team is not saying that the members

will not do the work. e team is merely saying that its

members cannot do more work and is letting the businessdecide whichwork must be done. At the end of the sprint,

all the work the business selected will be delivered.

Th Pjct Scai

Well summarize the key points of this article by using a

practical example: a request that came in to our BI team

from within an insurance organization where I function

as BI team lead (product owner).


22/60


AGIle bI

e following user statement represents the request:

As the head of the Motor Insurance Division

I want to have a report that provides the average cost

of claims per branch for payments made to towing

service partners for the past three years

In order to set the benchmark target and to track

progress against it in the future, with the potential of

saving the company millions of dollars for handling

service partner payment agreements.

Let us follow the agile steps.

First, the entire BI team meets at their sprint groom-

ing session where the new request is presented. After

reviewing the captured user statement (use case), the team

agrees that although this seems on the surface to be a

simple BI application (report) request, it is not. e main

issue is that the current data warehouse does not contain

a transactional claim payment fact table, which would

need to be referenced in order to determine who the

vendor was on the payment record.

Using the vendor categorization attribute from the

dimension table, it would be possible to determine the

reason for the payment (e.g., towing service).

Because this request cannot be contained within a single

story, a new epic is created called motor cost of claims.

e following stories with agreed points are created from

this single request:

1. Source system analysis and profiling to determinewhere to get the claim payments data from = 20 points

2. Report design specification with explained calcula-

tions = 1 point

3. ETL source to staging load package = 13 points

4. ETL staging to ODS (operational data store) load

package = 13 points

5. Dimensional model design = 3 points

6. ETL ODS to data mart load package = 13 points

7. OLAP cube to load of data mart = 5 points

8. ETL full historic load = 8 points

9. ETL testing, reconciliation, and balancing to source

system = 8 points

10.BI application motor cost of claims = 5 points

11.User acceptance and sign-off = 5 points

Total Points: 94

Because there are only two BI developers and a single

BI analyst available to form the scrum team required for

this epic, the team decides that the full-time effort until

completion would be two sprints (each two weeks in

duration).

Sprint 1 will consist of stories 14 and Sprint 2 will

consist of stories 511.

e scrum team meets for their planning session, where

designs and solutions are proposed. Some stories are

broken into tasks so more than one person can work on

completing the story.

Sprint 1 is started, and for the next two weeks, daily

stand-ups are held first thing in the morning, giving the

team a sense of progress and helping them highlight their

impediments and request assistance where needed. e

scrum master diligently follows up with items raised bythe team.

Sprint 1 is completed on time (of course) and a review

session is set up for the scrum team, including an

invitation to the business user. e product owner then

demonstrates stories 1 through 4 in the session, even

showing the user how they are now able to query the

required data. An initial analysis is produced to highlight

a possible cost of motor claims to the user, who states


23/60


AGIle bI

that the numbers look fine but seem a bit higher than

expected. is is noted for Sprint 2.

A retro session is set up for the scrum team. ey share

the good, the bad, and the ugly aspects of Sprint 1 and

note lessons learned to apply during Sprint 2.

Sprint 2 is started, and for the next two weeks, daily

stand-ups are held first thing in the morning, giving the

team a sense of progress and helping them highlight their

impediments and request assistance. e scrum master

diligently follows up with items raised by the team.

Sprint 2 is not completed on time because the users didnot make themselves available during the final Story 11

user acceptance and sign-off. Unfortunately, by this time,

more requests have been received by the BI team and

additional epics and stories have been allocated to scrum

teams and sprints. e product owner, however, raises

with the motor cost of claim epic business owner that

Story 11 consisted of 5 points. ere is a story in the next

sprint (although it is for the financial analysis depart-

ment) that consists of 5 points.

My advice to any BI team looking

to adopt an agile approach is to

understand the methodology first

and then relate it back to what will

work best for your environment in

practical terms.

e product owner requests that the head of the motor

division set up a meeting with the financial account

owner of the financial analysis epic. In this meeting, it

is agreed by the business that Sprint 1 for the financial

analysis epic will swap out its story of 5 points to enable

the scrum team to complete the user acceptance and

sign-off story of 5 points for the motor cost of claims epic.

Th Qsti f Sppt ad maitac

As with everything great engineers build, there must be a

system to manage support and maintenance. Is it possibleto use an agile approach for this purpose, even though

support and maintenance are not typical projects?

e answer is yes. e key is to create a shadow sprint

called support that runs in parallel with each project

sprint. As each project sprint is finished, the support

sprint must also be closed and a new one created. is

coexistence of project and support sprints will enable

team members who finish their tasks sooner to move

directly into work in the support sprint; the product

owner with the team doesnt have to add more stories andincrease the scope of the current project sprint.

Using this approach will enable reporting of statistics

such as project burn-down and story velocity. For the

support sprints project, burn-down is irrelevant because

the scope of the sprint is not protected by the product

owner. However, because the support sprint is closed at

the same time as the project sprint, reporting on story

velocity will provide keen insight into support-response

times. Because none of us lives in an ideal world in which

everyone only does a set and assigned task, the BI teamleader/manager needs to perform a careful juggling act to

ensure projects are given priority and regular, day-to-day

operational support is not left out.

Th Diy

As with most tasks, there are many ways to produce

the desired result, yet the reasons for adopting an agile

approach in the BI environment are compelling. Some

reasons include the team dynamics of trust, ski ll,

cooperation, and motivation. e business is in control of

what work gets done and when, and the BI teams reputa-tion for successful delivery on time and within budget is

preserved or enhanced.


24/60


Editorial Calendar and

Instructions for Authors

InSTruCTIonS For AuTHorS

e Business Intelligence Journalis a quarterly journal that

focuses on all aspects of data warehousing and business

intelligence. It serves the needs of researchers and prac-

titioners in this important field by publishing surveys of

current practices, opinion pieces, conceptual frameworks,

case studies that describe innovative practices or provide

important insights, tutorials, technology discussions, and

annotated bibliographies. eJournalpublishes educa-

tional articles that do not market, advertise, or promoteone particular product or company.

editia Tpics

Journalauthors are encouraged to submit articles of

interest to business intelligence and data warehousing

professionals, including the following timely topics:

Agile BI

Architecture and deployment (including cloud com-

puting, software-as-a-service, Hadoop, MapReduce)

BI adoption and use

BI and big data

Data analysis and delivery

Data design and integration

Data management: MDM, data quality, and

data governance

Data warehouse and database technologies

Mobile BI

Project management and planning

Selling and justifying the data warehouse

editia Accptac All articles are reviewed by theJournalseditors before

they are accepted for publication.

e publisher will copyedit the final manuscript to

conform to its standards of grammar, style, format,

and length.

Articles must not have been published previouslyeither online or in printed form. Submission of a

manuscript implies the authors assurance that the

same work has not been submitted elsewhere, nor

will be submitted elsewhere during theJournal s

evaluation.

Authors will be required to sign a release form before

the article is published; this agreement is available

upon request (contact [email protected]).

eJournalwill not publish articles that market,advertise, or promote one particular product

or company.

Sissis

For more information and complete submissions

guidelines, please visit tdwi.org/journalsubmissions.

Materials should be submitted to:

Jennifer Agee, Managing Editor

E-mail: [email protected]

upcig Sissis Dadis

Volume 19, Number 3

Submission deadline: May 16, 2014

Distribution: September 2014

Volume 19, Number 4

Submission deadline: August 8, 2014

Distribution: December 2014


25/60


THe bI SmArT mACHIne

Ty Hitadis a technology strategist

for Idaho National Laboratory.

[email protected]

Watson and Siri:e Rise of the BISmart MachineTy Hitad

Astact

Th past fw yas hav sn a signicant vutin in

huan-cput intactin. Th a f sat achins is

upn us, with autatin taking n a advancd than

v f and pating aas that hav taditinay nuniqu t huan intactin. This vnt has th ptntia

t fundantay at th way usinss intignc (bI) is

xcutd and dpyd acss industis as w as th bI

ay pay in a aspcts f dcisin aking.

Wats: Ha ss machi i a batt f

lgica Spacy

In 2011, at the commencement of a special episode of

Jeopardypitting man against machine, host Alex Trebek

indicated that you are about to witness what may prove

to be an historic competition. He was right.

In this competition, IBM Research charged forward

to take the next step in the evolution of computational

leadership. is was the follow-up to a groundbreaking

1997 chess match in which IBMs supercomputer Deep

Blue faced off with Garry Kasparov, chess grandmaster.

at contest proved that a supercomputer could apply

programmatic logic to outperform a human master, in

this case at the game of chess.

With this new competition, IBMs team was faced withprofound and new levels of challenges. With chess, there

are predefined rules of movement. Deep Blue focused on

analyzing a ll of the possible outcomes and probabilisti-

cally determining the most optimal next move to counter

its challenger. It took into consideration patterns associ-

ated with Kasparovs past play along with the patterns of

many other great chess players.


26/60



eJeopardychallenge was inherently different. It

required that the machine think like a human and

interpret language like a human. To up the ante, IBMdidnt take on just anyJeopardycompetitors in their

demonstration of computing excellence; they took on

Brad Rutter and Ken Jennings, the two most successful

champions who had ever played the game. e bar was

set high; the team needed to develop a system that would

interpret, solve, and respond to clues that spanned many

topics presented in various formats.

To accomplish this, researchers developed Watson using

natural language processing and text analytics to develop

the basis for the human-computer interaction layer as wellas a probabilistic approach to identify the best answer

for each specific clue. e result displayed not only the

answer but also a graphical representation showing the

top three potential answers and the probability of being

correct. Unlike the chess match, which allowed Deep

Blue the traditional chess timing rules to do its analysis

and return a result, Watson was under a time crunch to

perform all of its computational processing more quickly

than its two all-star competitors.

With the complexity of finding the right answer forthe given clue paired with the relative messiness of

human language inherent within the clues themselves,

Watson proved that even with high-power systems and

engineering genius on the back-end, it was still a complex

challenge. Watsons performance was not without quirks,

resulting in a tie at the end of the first day of play.

By the end of the three-day exhibition, Watson came

out on top, earning more than $77,147 compared to the

$24,000 and $21,600 of its competitors. Ken Jennings,

ever a good sport, bowed to the newJeopardychamp. Ifor one welcome our new computer overlords, he wrote

on his video screen, quoting an episode of e Simpsons.

Sii: bigig Aticia Itigc t th Cs

On October 4, 2011, Apple raised the stakes in the

battle for mobile supremacy with its launch of Siri on

the iPhone 4s. is innovative feature distinguished the

iPhone 4s from its competitors and laid the groundwork

to become the digital personal assistant of the future. Siri

provided a mechanism for end users to push a button and

ask a question, which would then be processed against

a multitude of applications on the device (includingreminders, calendars, messaging, e-mail, notes, music,

clocks, maps, and Web browsers) to either perform a

function or return related content. is elevated the

mobile phone from a portable computer to a personal

digital assistant, freeing the user from needing to know

which application should perform the requested function.

Now users could speak a simple command and have the

phone perform the majority of the processing needed to

respond accordingly.

e base technology supporting Siri was importantbecause it went beyond simply doing speech recognition

to execute a command. It married recognition with

natural language understanding to determine what type

of action the end user intended, identify the relevant

functionality, execute the command, and return a

response within the context of the request. is was

significant because it was built into a mobile phone

intended to be carried around and provide on-demand

access wherever and whenever the end user needed

itthe embodiment of computational mobility. Watson,

although much more capable in terms of its processingpotential and its sub-second response time, required a

cluster of 750 computers with 2,880 processor cores on

10 server racks to function, which significantly limited its

portability.

Sat machis f bI

Watson and Siri have demonstrated that natural language

understanding has the potential to fundamentally change

how end users interact with computational processing.

ese same trends also have the potential to fundamen-

tally alter how users engage business intelligence systemsin the decision-making process.

Traditionally, business intelligence suites have focused on

search and navigation as the mechanism for providing

content to end users within a business systems repository.

Both of these focus on metadata attached to predefined

reports and dashboards. is metadata includes report

titles and descriptions, but it is limited in providing a way

to find specific answers to questions. is is where natural


27/60



language understanding bridges the information gap to

support business intelligence. Instead of end users typing

sales into a search bar and needing to know whetherthey want the report sales by date or the report sales by

market segment, users would prefer to type exactly what

theyre looking forWho has sales growth of 10 percent

or more? or List sales growth at least 10 percentand

have the engine display a dashboard of sales filtered to

show only those segments of the business that have sales

of 10 percent or greater. Taking it one step further, users

would like to dictate their search requests, no keyboard

required.

Complicating the engines job is that the users engage-ment is context sensitive. Unlike the search engine, which

can index report metadata in the same fashion for every

company, a natural language engagement requires much

more context about the content to be effective. For Siri to

be effective at answering questions, it must interact with

multiple distinct content stores such as maps, calendars,

the Web, and so on. e engine has to determine the

most probable purpose of the engagement and invoke the

mechanism to call that functionality.

Business intelligence tools of

the future can learn from Siris

simplicity as they strip away the

complexity associated with knowing

how and where to find information

and provide a simple and universalinterface for users.

To succeed in applying natural language understanding

and advancing human computer interaction for business

intelligence, engines must address three aspects of the

problem: consumption, understanding, and response.

ese three represent the input, processing, and output

stages of system design.

Cspti

As organizations move into this new paradigm, the first

area to address is request consumption.

e traditional method of end users interaction with a BI

suite is to type one or more terms related to the request

or to navigate to a predefined location where known

information is located. is usually requires multiple

steps and in some cases requires end user training to

ensure that users understand how to move through the

business systems.

To simplify this interface on the iPhone, Apple intro-

duced an advanced voice-to-text system that takes a

request in the form of human speech and translates it

into a string of text that characterizes the request. Apple

removed the barriers of training and complex system

interaction by boiling down the interface to the single

push of a button. In response to this simple action, Siri

is ready to accept any command that the user desires.

is opens a conversation stream between human and

machine.

Business intelligence tools of the future can learn from

this simplicity as they strip away the complexity associ-

ated with knowing how and where to find information

and provide a simple and universal interface for users. e

BI tool must facilitate a request in the language of choice

and have the system perform the heavy liftingconvert-

ing the request into a set of systematic processes that will

supply the users desired objectives.

Watson didnt use voice-to-text processing, but insteadhad the clues fed to it at the same time that Alex Trebek

read them to the other competitors. is is similar to the

way end users naturally inquire with respect to questions

about business analytics. It is much more familiar for

an end user to ask, What is happening to the sales in a

certain region since we started our marketing campaign?

than to formulate a complex SQL (structured query lan-

guage) statement, a MDX (multidimensional expressions)


28/60



statement, or visit a series of screens that will dynamically

create the back-end SQL or MDX statement(s).

It is not optimal for an executive with a question to

navigate to a portal and navigate to the right report

to answer questions. Many organizations have created

business intelligence competency centers where executives

can send a request and have a team of BI analysts extract

and return an answer.

As technology advances in natural language understand-

ing, the process of engaging an analyst to research the

question and provide an answer could become a thing of

the past. e executive could e-mail or send a text mes-sage to a virtual assistant directly and the system would

interpret the objective, perform the analysis, and return

an answer without the delay of human intervention.

With more public-facing business intelligence solutions

enabling customers to perform self-service information

gathering, this concept can be extended to social media

venues. As questions or requests are made through

Facebook, Twitter, Instagram, or a myriad of other social

media or communication platforms, these requests can be

parsed and their context identified; the request can thenbe answered without needing a human customer service

representative.

is evolution in how requests are consumed and

fulfilled will fundamentally change how businesses will

work in the future and will have a dramatic impact on

the economy as a whole. In the fall of 2013, Gartner

predicted that the rise of smart machines will have a deep

and widespread impact on businesses through 2020. is

prediction includes the potential widespread elimination

of millions of middle-class jobs; those employees focusedon providing this middleman service may be replaced

by smart machines (Gartner, 2013). Although this has

serious implications for the state of the economy as a

whole, it also means that companies that can get in front

of the wave and power the coming evolution will win in

the end.

udstadig

e greatest technical challenge comes after a user makes

a request. is includes bringing a level of context andunderstanding through elements of natural language

processing. Language is messy; the same fundamental

request can be conveyed in multiple ways, using different

words and phrases and through various communication

channels. Add the global connectedness of business

transactions and the need to support multiple languages

and dialects, and the challenge increases dramatically.

To overcome this, natural language processing doesnt try

to develop a prescriptive set of rules to follow under every

circumstance, but instead uses machine learning andstatistical probability to find patterns of speech and likely

meanings.

e first step in natural language understanding is

taking a string of characters and determining how to

break it into parts that can be used to drive processing.

ese pieces, whether words or phrases, are the basis for

interpreting the request.

Parsing a sentence can be as simple as identifying where

the spaces are in a sentence and breaking the sentenceat these spaces. As punctuation is factored in, the

process becomes more challenging. When evaluating a

period, the parser needs to distinguish between multiple

instances of usage. When a period falls at the end of a

sentence, it is not attached to a word but to a sentence and

has no relevance to the adjacent word. If it is attached to

a word inside a sentence (e.g., Mr. or Dr.), it is associated

with the word and not the sentence; it might or might

not be able to be stripped away without changing the

meaning of the word. If the period falls inside the word,

stripping it out might have a more significant impact.For example, U.S. (with periods) represents a country,

whereas without periods it is a pronoun representing the

speaker and others.

ese complexities apply to other punctuation characters

as well. Capitalization can be used to help define sentence

boundaries but brings similar challenges in the form of

acronyms, mixed case names, and other use of capital

letters that are not the norm.


29/60



e next step in natural language processing involves

identifying the parts of speech of the identified words.

Different parts of speech distinctly affect the meaningof the sentence. Nouns represent the entities of concern,

adjectives are used to provide additional context to the

nouns, verbs are often associated with the action, and

prepositional phrases provide context to the sentence as

a whole.

Parsers often rely on predefined corpuses of text that have

been hand annotated by experts and help define statistical

models for determining the part of speech of specific

words in specific positions. ese corpuses are used to

train models that can be applied to an unknown set oftext to determine the most probable part of speech for

each word in the sentence.

Words are often based on the same root and

multiple forms have closely related meanings. Plurals

(e.g., tree/trees, ox/oxen, sheep/sheep), gerunds (e.g.,

water-ski/water-skiing, write/writing, find/finding), and

other grammatical vehicles take a word with very similar

meaning and mask it so that it fits into a sentence in a

different way. Comparing these words in their raw format

could miss the fact that the words are meant to achievethe same request. Natural language processing can

identify the common root among the different variations

of a word, thus identifying the wo

Documents

Business Intellegence Journal