Analytics Julyaugust 2012

8/11/2019 Analytics Julyaugust 2012

1/47

h t t p : / / w w w . a n a l y t i c s - m a g a z i n e . O R g

ALSO INSIDE:

July/august 2012DRiving BetteR Business DecisiOns

T e F t e of Fo ecastini r o o d ,

r o o r

Anal tics & Bi Datas . : B or

r d d o d o - r

Anal tics at IBMt o o , o o ,r or d d

COMpETITIvECrOwDSOurCINgANALyTICS AS SpOrT:

A wIN-wIN FOrCONTESTANTS AND

COrpOrATIONS

exar ceOp Jo r


2/47

Bellevue, WA, USA Haarlem, the Netherlands Singapore Shanghai, Chinawww.aimms.com [email protected]

AIMMS is a registered trademark of Paragon Decision Technology B.V. Other brands and their products are trademarks of their respective holders

The AIMMS PRO PlatformA New Revolution In Optimization

Ever thought about...creating an optimization App Store

for business users?

... providing web access to your optimization solutions?

...deploying in the cloud?

No? We have!

Check out our AIMMS PRO Platform at:

www.aimms.com/aimms-pro-platform

1 | a n a l y t i c s - m a g a z i n e . o r g

t or o d i n s i D e s t O R y

How many analysts does it take tosolve a problem?

That may sound like the start of a bad joke, but no one was laughing in 2006when Net ix offered $1 million to any -one who could come up with a collabora-tive ltering algorithm that improved theperformance of Cinematch (Net ixs in-house software) by at least 10 percent.Cinematch predicts which movies Net ixcustomers like and makes movie rec-ommendations to customers based onthose predictions. The goal: boost cus -tomer satisfaction and retention alongwith sales.

Three years later, after receiving sev-eral thousand entries from more than 100countries, a winner was announced, the$1 million prize was awarded and a cot -tage industry online marketplaces forbusiness projects where companies postchallenges, provide data and offer prizesfor the best solutions took off.

While Net ix reportedly performedno formal cost-bene t analysis on theNet ix Prize, the company was clearly

pleased with the results. At the time,Net ix CEO Reed Hastings said, Youlook at the cumulative hours and youregetting Ph.D.s for a dollar an hour.

In this months cover

story , MargitZwemer, a data scientist and commu-nity manager at Kaggle, provides an

inside look at crowdsourcing theconcept that turns complex analyticalproblems into a competitive sport opento analysts, astrophysicists or anyoneelse who cares to submit a solution. AsZwemer notes in her article, the con-cept is not new; as far back as the 18thcentury, the British government offeredmore than 100,000 in prize money toanyone who could come up with simpleand practical methods for measuringlongitude to assist maritime navigation.

The Net ix Prize, however, helpedturn crowdsourcing into a modern-day,mainstream corporate strategy. Data

research competitions are a resource-ef cient way for organizations to solvecomplex data problems, and they cre -ate a meritocratic market for talent thatchanges the way analysts work, Zwe -mer writes. Kaggle, an online platformfor predictive modeling and analyticscompetitions, was one of many compa-nies that jumped on the crowdsourcingbandwagon in the aftermath of the Net-

ix Prize. According to Zwemer, Kaggle

boasts a worldwide online communityof more than 40,000 data scientists andpredictive analysts, competing under theslogan making data science a sport.

peteR hORneR, eDitORr. or r@ . or .or


3/47

DRIVING BETTER BUSINESS DECISIONS

C o n t e n t s

FEATurESReseaRch as a cOmpetitive spORtBy Margit ZwemerCr wd urci g h full a aly ic valu chai , ar i g wi h hpr dic iv m d l i lf.

cOllaBORative fORecasting: fROm visiOn tORealityBy Brian Lewistra i i i g fr m p ri dic, i la d ac ivi i a i gl , r al- im

rpri pr c .

fORecasting an upwaRD tRenD?

By Jack Yurkiewiczsurv y f f r ca i g f war r v al i r i g r d a d

w d v l pm .

analytics & Big Data: skeptics vs. enthusiastsBy Gary CokinsAcc p a c f a aly ic ackl i u r la d big da ag i g hr ugh gr wi g pai .

pReDictaBility Of time seRiesBy Subir Mansukhanis a i ical m a ur i f r h l v l f dif cul y i ch i g a

appr pria m d l.

cORpORate pROfile: iBmBy Arnold Greenland t ch l gy, c ul i g gia b a a l g, rich hi ry wi hadva c d a aly ic .

12

16

20

25

29

32


Discover the easiest, fastest, lowest cost and risk onramp to full-power data mining and predic veanaly cs: Analyze your data in Microso Excel withXLMiner so ware, now from Frontline Systems.

You wont need a sta s cs PhD, months of learningme, or a huge budget for enterprise data mining

so ware. But you will have the power of predic veanaly cs and data visualiza on at your nger ps.

Sophis cated Data Mining Power.

XLMiner goes far beyond other sta s cs and forecas ngadd-ins for Excel. It starts with mul ple regression,

exponen al smoothing, and ARIMA models, but goesfurther with regression trees, k-nearest neighbors, andneural networks for predic on . It o ers discriminantanalysis, logis c regression, k-nearest neighbors,classi ca on trees, nave Bayes and neural nets forclassifca on , and associa on rules for a nity analysis.

Prac cal Data Cleaning and Explora on.

XLMiner includes u li es for data sampling andpar oning, missing data handling, binning, and

transforming categorical data. You can apply principalcomponent analysis, k-means clustering, andhierarchical clustering to simplify and cluster your data.

Help, Textbook and Instructor Support.

XLMiners extensive online Help introduces the datamining methods and explains the op ons in each dialog.With the popular textbook Data Mining for BusinessIntelligence built around XLMiner so ware you cango from beginner to expert business analyst in a high-demand eld. Instructors can take advantage of low-cost classroom licenses, PowerPoint slides, example

les, solu ons to exercises, an online forum, and more.

Download Your Free Trial Now.

Visit www.solver.com/xlminer to learn more, registerand download a free trial or email or call us today.

Tel 775 831 0300 Fax 775 831 0314 [email protected]

XLMINER: VISUAL DATA MININGWorks with PowerPivot to Access 100 Million Rows in Excel

25

20

12

JULY/AUGUst 2012Brought to you by


4/47

DRIVING BETTER BUSINESS DECISIONS Brought to you by

The Excel Solvers Big Brother Makes Op miza onEasier, More Powerful and Far More Visual.

From its classic Solver Parameters dialog to its Ribbon,Task Pane, charts to visualize the shape of func ons,Constraint Wizard, model diagnosis, Help, and coveragein leading management science textbooks, Risk SolverPla orm makes it easier to learn and use op miza on.

Learn and Use Monte Carlo Simula on, DecisionAnalysis, and More with the Same So ware.

You can use its powerful op miza on features alone but Risk Solver Pla orm is also a full-power tool forMonte Carlo simula on and decision trees, with a Dis-tribu on Wizard, 50 distribu ons, 30 sta s cs and riskmeasures, and a wide array of charts and graphs.

Use the Full Spectrum of Op miza on Methods.

Fast linear, quadra c and mixed-integer programming is just the star ng point in Risk Solver Pla orm. SOCP,nonlinear, non-smooth and global op miza on are just

the next step. Easily incorporate uncertainty and solvewith simula on op miza on, stochas c programming,and robust op miza on all at your nger ps.

Go Beyond Single Op miza on/Simula on Solu ons.

De ne parameters in seconds, run high speed mul -wayparameterized simula ons and op miza ons, captureall the solu ons, and produce instant reports, chartsand graphs, without programming. And you can doeven more in Excel VBA.

Free Trials and Support for Your E orts at Solver.com.

Download a free trial of Risk Solver Pla orm, and learnabout the many other ways we can support yourlearning and teaching e orts all at www.solver.com

TEACH AND LEARN OPTIMIZATION AND SIMULATION

RISK SOLVER PLATFORM

Tel 775 831 0300 Fax 775 831 0314 [email protected] | a n a l y t i c s - m a g a z i n e . o r g

REGISTER FOR A FREE SUBSCRIPTION:http://analytics.informs.org

INFORMS BOARD OF DIRECTORS

President Terry P. Harrison, Penn State University President-Elect Anne G. Robinson, Verizon Wireless

Past President Rina R. Schneur,Verizon Network & Technology

Secretary Brian Denton,North Carolina State University

Treasurer Nicholas G. Hall, Ohio State University Vice President-Meetings William Bill Klimack, Chevron Vice President-Publications Linda Argote, Carnegie Mellon University Vice President- Sections and Societies Barrett Thomas, University of Iowa Vice President- Informati on Technology Bjarni Kristjansson, Maximal Software Vice President-Practice Activities Jack Levis, UPS Vice President-International Activities Jionghua Judy Jin, Univ. of Michigan Vice President-Membership

and Professional Recognition Ozlem Ergun, Georgia Tech Vice President-Education Joel Sokol, Georgia Tech Vice President-Marketing,

Communications and Outreach E. Andrew Andy Boyd,University of Houston

Vice President-Chapters/Fora Olga Raskina, Con-way Freight

INFORMS OFFICESwww.informs.org Tel: 1-800-4INFORMS

Executive Director Melissa Moore

Meetings Director Teresa V. Cryan Marketing Director Gary Bennett Communications Director Barry List

Headquarters INFORMS (Maryland) 7240 Parkway Drive, Suite 300

Hanover, MD 21076 USA Tel.: 443.757.3500 E-mail: [email protected]

ANALYTICS EDITORIAL AND ADVERTISING

Lionheart Publishing Inc., 506 Roswell Street, Suite 220, Marietta, GA 30060 USATel.: 770.431.0867 Fax: 770.432.6969

President & Advertising Sales John Llewellyn

[email protected] Tel.: 770.431.0867, ext.209 Editor Peter R. Horner [email protected] Tel.: 770.587.3172 Art Director Lindsay Sport [email protected] Tel.: 770.431.0867, ext.223 Advertising Sales Sharon Baker [email protected] Tel.: 813.852.9942

the Institute for Operations Research and the ManagementSciences (INFORMS). For a free subscription, register athttp://analytics.informs.org. Address other correspondence tothe editor, Peter Horner, [email protected]. Theopinions expressed in Analytics are those of the authors, anddo not necessarily re ect the opinions of INFORMS, its of cers,Lionheart Publishing Inc. or the editorial staff of Analytics. Analytics copyright 2012 by the Institute for OperationsResearch and the Management Sciences. All rights reserved.

DEpArTMENTSi d s or

ex ed

pro c r

a t !

for

f -m a

t a

co r pr

l r R o r

v o

1

4

6

7

10

35

37

41

43

45

37

41


5/47

w w w . i n f o r m s . o r g

Predictive analytics is a powerful methodused by leading business organizations to pre -dict future events and behavior in order to op-timize current marketing, product, operationsand sales actions. The prediction is based on

a basic fundamental: past behavior predictsfuture behavior. As an example, lets take an e-commerce

company selling furniture online. The onlinesite has about a 10 percent contact rate, i.e.,of the 100 people who come to the website,about 10 people contact customer servicewithin 24 hours. The VP of Customer ServiceOperations is looking to reduce the contactrate as each contact costs the company incre-mental dollars in operations. Lets say a 1 per -cent decrease in contact rate would amount toa savings of $1 million a year for the company.The operations team decides to use predictiveanalytics to understand the drivers of contact.

By using certain predictive analytics tech-niques on historical data, a relationship isidenti ed between help page visits and visitorsmaking a phone call to the customer service

department. Speci cally, 50 percent of visi -tors, after having gone through three distincthelp pages, call customer service. This is avery helpful clue. If the visitor can be inter -cepted before he hits the third help page either

by providing better help content or a live chator a clari cation window based on what he isbrowsing, contact rate can likely be reduced. Ihave seen organizations save and make mil -lions by understanding this kind of pivotal rela-tionships between behaviors and events.

On the ip side, predictive analytics, in spiteof being a powerful optimization technique, isoften left to the devices of data miners anddata scientists, and thus is often misunder-stood and misused by businesses. Having a

rm hold on what predictive analytics is not willmake predictive analytics a more useful toolfor businesses.

1. Predictive analytics is not new. Somenews items such as the Feb. 16 New YorkTimes report on retail company Targets predic -tion of teenage pregnancy implies that predic-tive analytics is a new found technique. But infact predictive analyt ics is not new. Fischer andDurand, founders of the Econometric society,built one of the rst credit scoring models 80years ago. But predictive modeling techniquesgo back thousands of years Indian astrologi-cal charts used to arrange marriages are onesuch example.

2. Predictive analytics does not produceperfect predictions. Often while building the

model, it is clear to all that model prediction hasa probability associated with it, but upon suc-cessful use, there is often a misplaced senseof perfectness in the scores. As in the case ofour e-commerce company, 50 percent of visi -tors after visiting three help pages are goingto call, but the other 50 percent wont call. Themodel predicts by maximizing the likelihood,and a certain degree of misclassi cation al -ways exists. By using other predictors, theseodds can be improved, but the prediction willstill not be a 100-percent accurate.

3. A good software tool does not mean agood model. With tremendous develop-ment on the software tool front with better GUI

as well as higher automation, people new tothe eld often mistakenly believe that a goodmodel can be automatically built by pressinga build model button. A good model requiresproper technical skills and a proper model-building process. Surprisingly, sometimeseven proper skills and a proper process doesnot deliver a good model with decent lift andlow misclassi cation.

4. A good model does not always meanbetter business results. This is one of thosehighly prevalent myths that even experiencedanalysts fall for, leaving them frustrated whennobody in the business seems to care for theamazing model they built. A good model gen -erates business impact only when the rightstakeholders are brought into the analyticsprocess at the right time, thus building proper

4 | a n a ly t i c s - m a g a z i n e . o r g a n a ly t i c s | J U ly / a U g U s t 2 012

w r d o

By piyanka Jain

pr d , o b o r

o o q , o o d

o d r d d

, d o d r ood d

d b b .

e X e c u t i v e e D g e


6/47


alignment toward actionability using aframework. In our e-commerce exam -ple, if instead of building a contact rateprediction model, we had built a modelto predict visitors most likely to do live-chat, would the VP of Operations carefor the model and the results? Prob -ably not. Unless we can show the re -lationship between what we (predictivemodel/team) are trying to do and hisbusiness goal (contact rate reduction),the VP of Operations is not going to usethe model.

5. Models cant be built and forgot-

ten. Models become stale over time.If not maintained, they often stop de-livering the incremental value it start-ed with. As organizations embark onthe journey of competing on analytics,they need to be aware that it is not aone-time investment. You cant hire ex -ternal consultants, get the model builtand leave it at that. Model needs to betested, tweaked and then maintainedto continue delivering the incremental

bene t.

Piyanka Jain ([email protected]) founder, presidentand CEO of analytics training company Aryng, speaksregularly at business and analytics conferences ondata-driven decision-making in an organization. Her prior roles include head of NA Business Analytics atPayPal and senior marketing analytics position with Adobe. She is an INFORMS partner.

e X e c u t i v e e D g e

The Op miza on / Simula on Library that Speaks theLanguages of Desktop and Web Compu ng.

No other callable library matches the exibility ofSolver SDK Pla orm from solving Excel models onyour server, to solving models created in your browser.

Build Your Model Faster in Excel or Visual Studio.

Use Risk Solver Pla orm or its subsets to build yourmodel in Excel. Use Microso Visual Studio to createyour model in C++, C# or VB.NET. Use C, R, MATLAB,Java, PHP or JavaScript its your choice.

Use Op miza on and Simula on as a Web Service.

Run the Solver Server that comes with Solver SDK onany PC or server, then solve your models over thewire using our standard Web Service, SOAP and WSDLprotocols. Get real- me progress, or just disconnectand retrieve the solu on later its your choice.

Solve Your Most Challenging Linear Mixed-Integer,Nonlinear and Stochas c Op miza on Models.

Solver SDKs wide choice of built-in and plug-in SolverEngines handle every type and size of op miza onproblem, from Gurobi and Xpress-MP to Mosek andKNITRO. Use the SDKs built-in simula on op miza onand stochas c programming algorithms, and supportfor mul -core CPUs.

Download a Free Trial at www.solver.com.

Learn more and download a FREE trial version of RiskSolver Pla orm, Solver SDK Pla orm, and our eightplug-in Solver Engines from Solver.com

THE MOST FLEXIBLE WAY TO DEPLOY YOUR MODEL

SOLVER SDK PLATFORM

Tel 775 831 0300 Fax 775 831 0314 [email protected]

mod b o o r. i o d,

o o d r

r r d . aor o b r o

jo r o oo , d o

b r o o - .

s b r b oAnalytics

Its fast, its easy and its FREE!Just visit: http://analytics.informs.org/


7/47


8/47w w w . i n f o r m s . o r g

Been cleaning out the home of ce thismonth. Apparently, this is the kind of thing thatis done by a newly tenured professor with nosummer teaching or administrative responsi-bilities to avoid tackling the gnarly research

problems on his desk. In addition to the manylong-lost letters, books, papers, ticket stubs,non-disclosure agreements, unmarked CDs,forgotten photos, abandoned textbooks andstray postcards, Ive also managed to un -earth an impressive pile of discarded plas-tic lanyards, the kind one acquires at eventssuch as orientations, new student dinners andconferences.

In particular, the bright blue lanyards fromthe INFORMS Conference on Business Analyt -ics and Operations Research (BAOR) in 2011and 2012 caught my eye. Ive been going tothe fall INFORMS conference for many, manyyears [1], and its always a great pleasure tocatch up with familiar old friends there. But atthese past two BAOR conferences, the rstones that I have attended, I have also reallyenjoyed meeting interesting people who are

new to INFORMS. At the 2011 BAOR confer -ence in Chicago, I had my rst contact with ananalytics leader named Scott Friesen. At the2012 BAOR conference in Huntington Beach,Calif., I had a really interesting breakfast con -

versation with an auto industry pioneer namedSandy Stojkovski.

Inspired by the discovery of these old lan-yards, I recently caught up with Scott andSandy last week to answer a couple of simplequestions: Who are you? and How did youend up attending an INFORMS conference?Their responses were both fascinating andinstructive.

Scott Friesen is senior director of analyticswithin the Customer Insights Unit (CIU) at Best

Buy. This is the latest in a series of roles thathe has played within that company since nish -ing his MBA in 2004. Over the past two years,Scotts group has grown from 11 to more than30 professionals by consistently demonstrat-ing business value to its internal customers.Scotts CIU is viewed within Best Buy as the go-to place for executives and managers acrossthe company to understand customer behav-ior, pro tability models, competitive trends anda number of other data-driven questions of in-terest to executives, marketers, retail manag -ers and strategic planners.

The journey that led him to this role as ananalytics leader has been unique and can betraced at least as far back as his businessschool statistics class with Paul Glassermanat Columbia (he confessed to me that hefound Professor Glasserman and his course

to be inspiring). Along the way, hes workedin strategy for BestBuy.com, been trained asSix Sigma Blackbelt and Master Blackbelt,launched a new services business within theGeek Squad organization, and set up a claims

analytics function (complete with actuaries) tomanage risk in the companys extended ser -vice warranty business.

Sandy Stojkovski is president of Scenaria(www.scenaria.com ), a consulting rm that shefounded with AVL ( www.avl.com ) in 2010. Sce -naria develops sophisticated analytic modelsto provide unique insights and strategy guid-ance to its clients, who are typically executiveswithin the auto industry. Starting this companyis only the most recent time that Stojkovski

has broken the mold during her career in theauto industry. She has also been a 22-year-old green eld plant manager, gained exper -tise in value engineering while working for aparts manufacturer, and developed a globalpower train strategy for a prominent automo-tive brand. She also managed to complete agraduate engineering degree and an MBA be -fore turning 30 while working full-time.

Prior to starting Scenaria, Stojkovski de -veloped deep expertise with vehicle fuel ef -ficiency standards first while working on fueleconomy as an engineer at Ford and thenwhile launching and growing an energy ef-ficiency practice at an engineering servicesfirm called Ricardo. Her bold vis ion to trans -form the way that auto company executivesdo strategic planning has in turn requiredthe company to develop a management


B o r b do o

By viJay mehROtRa

n or infORmsco r o B

a d O r oR r d d d .

a n a l y z e t h i s !


9/47

w w w . i n f o r m s . o r g8 | a n a ly t i c s - m a g a z i n e . o r g a n a ly t i c s | J U ly / a U g U s t 2 012

science-based approach to manage-ment consulting to help its clients intelli-gently address the types of high-stakesdecisions that have to be made in thecapital-intensive automotive industry.The companys projects typically fea -ture a variety of analytic tools, includ-ing experimental design and simulationmodeling and visualization.

At rst, I found it curious that neitherScott nor Sandy had any degrees in math -ematics, statistics or operations research(though each of them appreciated what theyhad picked up in their respective academicprograms). While both expressed genuine

respect for the mathematical techniquesunderpinning this analytics revolution, nei-ther felt that their relative lack of formalmathematical training had handicappedthem professionally in any way.

Instead, what really stood out was thateach of them had an almost viscerallystrong inclination toward systems think-ing. While Friesen attributed this to hisundergraduate roots in the liberal arts,Stojkovski credits her engineering back -ground for this orientation toward seeingproblems in the context of the larger sys -tems in which they exist.

They also shared several other keytraits that are typical of successful leadersin the analytics industry. Throughout theircareers, the interesting variety of roles and

assignments revealed both of them to bevoracious learners, as well as smart, judi-cious risk takers. Friesen and Stojkovskialso both emphasized the value of buildingstrong relationships, with Friesen passingalong an important equation that a mentorhad shared with him (Trust = Credibility *Intimacy/Risk). Each of them repeatedlyspoke of looking at problems strategicallyand continually seeking to get closer to theroot of the problem because, as Friesen putit, thats where the leverage is.

This search for valued relationshipsand intellectual leverage is what led bothFriesen and Stojkovski to INFORMS andin particular to the increasingly popu-lar spring BAOR conferences. At the2011 BAOR conference, Friesen madea presentation [2] about leveraging tech -nology and relationships to implementanalytic capabilities in the complex anddynamic world of Best Buy, participatedin a lively panel session on Growing an

Analytics Capability, and enjoyed the


Join the Communityfor inside information.

SECTION ON

ANALYTICS

Looking to interact and networkwith analytics professionals?

http://analytics.section.informs.org

w bo x r d r or

qd r

r o o , r r r o or

rd d d

ro o .

h pro o Analytics

Its fast and its easy! Visit:http://analytics.informs.org/button.html


10/47


mix of industry and academic conver -sations. Meanwhile, Stojkovski startedout reading Analytics magazine as partof her ongoing search for new knowl-edge and became intrigued with theopportunity to leverage lessons fromdifferent industries into Scenarias ap -proaches. Eventually she made herway to the 2012 BAOR conference inHuntington Beach, where she delivereda presentation [3] on a strategic plan -ning, analysis and a data visualizationplatform that her company had devel-oped for the California Hybrid, Ef cientand Advanced Truck Research Center

to help the organization to jointly evalu -ate technology and policy decisions.

These days, there is a great deal ofgeneral talk about the rapid growth andevolution of analytics. Friesen and Stojk -ovski provide two speci c and optimisticvisions for the future. Scott and his grouphave recently received management ap-proval for a new multi-million dollar cus-tomer information infrastructure that hebelieves will lead to a quantum growth inthe types of insights his group can pro-vide to its many business customers with-in Best Buy. For Sandy, the progressivetightening in the CAFE standards, whichare slated to increase by 80 percent by2025, means complex investment deci -sions and serious risks for the automotive

industry and great opportunities forScenarias industry expertise and analyt -ics capability to make an impact.

For my part, I feel fortunate to be ableto engage personally and professionallywith these folks (I was taught long ago tosurround myself with likable people whoare different from me and smarter thanI am). In particular, I am thankful to IN -FORMS for creating the BAOR confer -ence and thus establishing a compellingspace for hatching these types of newconnections. Moreover, meeting peoplelike Scott and Sandy has also convincedme that there are many different types of

analytic leaders out there and I cantwait to see more of them at the next springparty (April 7-9, 2013 in San Antonio).

We all have an awful lot to learn.

Vijay Mehrotra ([email protected]) is anassociate professor, Department of Finance andQuantitative Analytics, School of Business andProfessional Studies, University of San Francisco.He is also an experienced analytics consultantand entrepreneur and an angel investor in severalsuccessful analytics companies. He is a seniorINFORMS member.


General ChairRonald G. AskinArizona State University

Program Co-ChairsJohn W. FowlerEsma GelArizona State University

INFORMS ANNUAL MEETING

PHOENIX 2012Register Today!

INFORMATICSRISING

INFORMATICSRISING

October 14-17, 2012Phoenix Convention Center, Phoenix, AZ

Plus

INFORMS-SMDM Joint Programin conjunction with theSociety for Medical Decision Making

Special Joint ProgramWednesday, October 17

Followed by theSMDM Annual MeetingOctober 17-20

Thanks to Our Sponsors

http://meetings2.informs.org/Phoenix2012

NOTES & FURTHER READING

1. For the interested reader, heres a link to a storyabout my rst fall conference: www.orms-today.org/orms-12-03/frsomething.html .

2. A video of Scotts 2011 presentation is availableonline at www.informs.org/Community/Analytics/Videos/Analytics-Process-Presentations .

3. An abstract of Sandys 2012 presentation isavailable online at http://meetings2.informs.org/Analytics2012/analyticssustainability.html .


11/47


We all love the hard side of analyti cs. Whatwe often struggle with as analysts is what youmight call the soft side of analytics, which isalways more challenging than the hard stuff.Here are a few of the reasons why.

Many times, the problem is not insuf cient

data, defective data, inadequate data models oreven incompetent analysis. Often, the reasonthat better decisions are not made in less timeis that many companies of all sizes have some,if not many, managers and leaders who struggleto make decisions with facts and evidence . . .even when it is spoon-fed to them. One reasonis that regardless of functional or organizationalorientation, some executives tend not to be ana -lytically competent or even interested in analy-sis. As a result, they tend to mistrust any and alldata and analyses, regardless of source.

In other situations, organizations still discountrobust analysis because the resulting implica-tions require decisions that con ict or contrastwith tribal knowledge, institutional customs,their previous decisions or ideas that they ortheir management have stated for the record.Something to keep in mind is that at least some

of the analysis may need to support the currentthinking and direction of the audience that is an-alytically supportable if you want the audience tolisten to the part of your analysis that challengescurrent thinking and direction.

Understanding the context or the why? ofanalysis is fundamental to bene ting from it.However, there are times when the results of ananalysis can be con icting or ambiguous. Whenthe results of analysis dont lead to a clear, un -arguable conclusion, then managers or execu -tives without the patience to ask and understandwhy? may assume that the data is bad or, morecommonly, that the analyst is incompetent.

Perhaps the most dif cult challenge an or -ganization must overcome in order to raise the

level of its analytical capability is the natural hu-bris of senior managers who believe that theirorganizational rank de nes their level of unaidedanalytical insight. Hopefully, as we grow older,we also grow wiser. The wiser we are, the slowerwe are to conclude and the quicker we are tolearn. The same ought to be true for us as weprogress up the ranks of our organization, butsometimes it isnt.

If these are the reasons for the organiza -tional malady of failing to fully leverage ana-lytics to make higher quality decisions in lesstime, what is the remedy?

For the analyst, I recommend the following:1. Put yourself in the shoes of the decision-

maker. Try to step back from the detailsof your analysis for a moment and askyourself the questions he or she will ask.

2. Engage your decision-maker in the

process. Gather their perspective as aninput. Dont make any assumptions. Ask lotsof questions. They probably know things thatyou dont know about the question you aretrying to answer. Draw them out. Schedule

updates with the decision-maker, but keepthem brief and focused on essentials. Askfor their insight and guidance. It may provemore valuable than you think.

3. Take time to know, explore andcommunicate the why? of your analysis.Why is the analysis important? Why arethe results the way they are? To whatfactors are the results most sensitive andwhy? Why are the results not 100 percentconclusive? What are the risks and why do

they exist? What are the options?4. Make sure you schedule time toexplain your approach and the why?Your decision-maker needs to knowbeforehand that this is what you areplanning to do. You will need to put thewhy? in the context of the goals andconcerns of your decision-maker.

5. Consider the possible incentivesfor your decision-maker to ignoreyour recommendations and give himor her reasons to act on yourrecommendations that are also consistentwith their own interest.

6. A picture is worth a thousand words.Make the analysis visual, even interactive,if possible.

7. Consider delivering the results in Excel(leveraging Visual Basic, for example), not


w o d o o rd o

By aRnOlD maRk wells

ho r, r r r o b o

or b o . w r o do d o r, r b

o o , ror x o

o dd r d ?

d b d or, or o o ,

o .

f O R u m


12/47


just in a Power Point presentat ion ora Word document. In the hands ofa skilled programmer and analyst,amazing analysis and pictures canbe developed and displayed throughVisual Basic and Excel. Everyexecutive already has a license forExcel and this puts him or her face-to-face with the data (hopefully ingraphical form as well as tabular).You may be required to create aPower Point presentation, but keepit minimal and try to complementit with Excel or another tool thatactually contains the data and the

results of your analysis.

Frustration with your decision-mak-ing audience will not help them, youor the organization. Addressing themwhere they are by intelligently andcarefully managing the soft side ofanalytics will often determine whetheryou make a difference or contribute toa pile of wasted analytical effort.

Mark Wells ([email protected]) is a principal with End-to-End Analytics. For 20 years, hehas consulted with the management of global supplychains in the chemicals, medical devices, consumergoods, high tech, automotive and retail industries, aswell as the public sector. Wells holds an MBA (focusingin operations management and operations research)from Drexel University. He publishes asupply chainaction blog on supply chain and analytics, from whichthis article is adapted. He is an INFORMS member.

f O R u m

INFORMS CREDENTIALING

TASK FORCE

Terry Harrison (Penn State) Lisa Kart (Gartner) Bill Klimack (Chevron) Dave Leonhardi (Boeing) Jack Levis (UPS) Polly Mitchell-Guthrie (SAS) Scott Nestler (Naval Postgraduate School) Michael Rappa (NC State/Inst. Adv. Analytics) Anne Robinson (Verizon Wireless)

COMING APRIL 2013...

SPONSORED BY INFORMS

ANALYTICS CERTIFICATION

Members interested in becoming a Certified Analytics Professional (CAP) make plans now to take the profession's first general a nalytics certificationexam. Offered at 2013 INFORMS Conference on Business Analytics andOperations Research, April 7-9, 2013, Sa n Antonio, Texas.

Watch future issuesof OR/MS Today andAnalytics Magazinefor more details.

Read The Shapeof AnalyticsCertification inFebruary 2012 issueof OR/MS Today .

BENEFITS OF CERTIFICATION Advances your career potential by setting you

apart from the competition Drives personal satisfaction of accomplishing a

key career milestone Helps improve your overall job performance

by stressing continuing professional development Recognizes that you have invested in your

analytics career Boosts your salary potential by being viewed as

experienced analytics professional

ELIGIBILITY

BA/BS or MA/MS degree in related area At least five years of analytics work-related

experience for BA/BS holder At least three years of analytics work-related

experience for MA/MS holder Verification of soft skills/provision of

business value by employer

fr r o o rd o - d

o , oor or o .

addr r r b

d r o d o

o d rr o

d r or o r b o o d

or .

s b r b oAnalytics



13/47


Competition in analytics is afamiliar concept on an orga-nizational level. An analyt -ics gap exists between the

predictive haves and have-nots: an insur -ance company that accurately predicts a

new customers actuarial risk, a mortgagelender that better predicts the probabilityof default, a retailer that better predictsits churn rate, a social network its num-ber of followers or an ad platform its click-through rates. If any of these companies

can accomplish their predictions betterthan the rest of their industry, they gain adistinct competitive advantage survivalof the analytic ttest.

What happens when this same ap-proach is applied to the building of the

predictive models themselves? The riseof competitive research has shown thatas much or more value can be derivedfrom competition between the predic-tive analysts. In this article, well exam -ine the past results of making analytics

By maRgit zwemeR

C

p R e D i c t i v e m O D e l i n g


cro d o r .

R r o or

The rise of competitive research has shown that as much or more value can be derivedfrom competition between predictive analysts.


14/47


a competitive sport and see how this ap-proach is affecting the full analytics valuechain, from problem identi cation to anal -ysis to implementation, and even howanalysts might be recruited and compen-sated in the near future.

ThE hISTOry OF INNOvATION prIzES

We tend to think of crowdsourcing as acost-effective way to accomplish low skill,repetitive tasks, but a quick glance at thehistory of innovation prizes shows that opencompetition can achieve breakthrough re-sults in high value-added areas. The tra -ditional methods of allocating research

funding and corporate analytics budgetscan encourage a risk averse and cautiousmentality . . . and solutions that are goodenough but not great. The winner-take-allnature of prizes breeds a form of intelligentrisk-taking that is particularly good at solv-ing previously intractable problems. Essen -tially, it is a way of crowdsourcing genius.

As far back as the 18th century, theBritish government offered more than100,000 in prize money to anyone whocould come up with simple and practicalmethods for measuring longitude to assistmaritime navigation. A watchmaker, JohnHarrison, won the task. Then there wasthe very rst non-stop transatlantic ight in1927, which awarded Charles Lindberg, arelatively unknown aviator at the time, the

$25,000 Orteig Prize. In 2004, a privatelyfunded team led by engineer Burt Rutancaptured the $10 million Ansari X Prizefor becoming the rst non-governmentalorganization to launch a man into spacenot once but twice within two weeks usinga largely reusable spacecraft. The 2009Net ix Prize demonstrated that this modelcan be applied equally well to algorithmicinnovation. Net ix offered a million dollarsfor a 10 percent improvement in its movierecommendation algorithm. An estimated80 percent of the movies that customerswatch on Net ix are found through therecommendation engine, so, given the

number of Net ix customers, the million-dollar prize was money well spent.

All of the prize contests mentionedhere produced remarkable accomplish-ments, and while they could have orwould have been achieved without suchbounties to spur them on, it is unlikelythey would have happened so quickly.

MODErN INNOvATION prIzES

Using innovation prizes to not onlysolve a one-off problem but as a regularpart of doing business is catching on inmany different areas. Topcoder is a well-known community of more than 400,000developers who compete in challengesranging from architecting an entire soft-ware system to creating a Web site to

developing new algorithms. The commu -nity competes to design and then buildeach module of the product, with a prizelinked to each component based upon itsdif culty. Innocentive is another innova -tion platform that hosts an even wider ar-ray of challenges, from nding biomarkersfor ALS (Lou Gehrigs disease) to invent -ing a low-cost rainwater storage system.From brainstorming, to theory, to blue-prints, to code, all of these can now becrowdsourced.

Before I dive more deeply into re-search as a competitive sport, I shouldtell you that I work for Kaggle, a predic-tive modeling competition platform. If theexamples I bring up seem to tilt heavilytoward my own organization, its becausethis is the area of competitive research Iknow best, and for which I can marshalhard evidence to back my claims.

Inspired by the Netflix Prize, Kagglewas founded in 2010 to apply the innova -tion prize model and digitize it, creating

c R O w D s O u R c i n g

For many years, INFORMS has offered a Job Placement Service to connectemployers and qualified O.R. and analytics professionals. This service can beused alone or in conjunction with the Annual Meeting Job Fair at the 2012 AnnualMeeting. Both give applicants and employers a convenient venue for meeting. JPSis free to INFORMS member applicants.

http://jps.informs.org

LOOKING? HIRING?

For info on job fair at the 2012 INFORMS Annual Meeting,

visit http://meetings.informs.org/phoenix2012/jps.html


15/47


a marketplace for data science. Kagglehosts prediction competitions that solvelarge-scale data proble ms in areas suchas business, health, education and sci-ence. The competitions lead to moreaccurate algorithms for the companiesthat sponsor them because they pit awide range of solutions and techniquesagainst each other on a data scienceproving ground. The online communi -ty has grown to more than 40,000 datascientists and predictive analysts, com-peting under the slogan making datascience a sport.

While the painstaking work of building

a great predictive model could not seemfurther from the mud and sweat tradition-ally associated with sports, the competi-tive dynamic between Kaggle participantsis what has consistently produced modelsthat are not just better than the currentstate of the art, but better than even theanalysts involved thought possible.

Kaggle competitions have dramati-cally outperformed pre-existing bench -marks in every competition run. Forexample, the best performing algorithmin Allstates Kaggle competition pro -duced a 271 percent improvement overthe internal version and an 83 percentimprovement over the expected best-case development of a similar modelusing internal or traditional third-party

resources. Allstate has validated thatthe Kaggle competition produced amodel that performed substantially bet-ter than what they would have expectedto develop internally or with a special-ized outside predictive modeling con -sulting firm. In a competition for anotherclient, they allowed their internal ana-lytics team to compete anonymously inthe competition. While they were notthe overall winners, their performancewell exceeded the benchmark that theythemselves had built before the compe-tition started.

The Kaggle platform also includes a

real-time leaderboard associated witheach competition. Getting instant feed -back on the accuracy of their modelsagainst a hidden validation set, rather thansubmitting models that are only judged atthe end of the competition, encouragescontest participants to push themselvesharder in order to leapfrog their competi-tors. The competitive dynamic drives datascientists to continue exploring ways toimprove predictive accuracy, spurred onby the knowledge that someone else hasfound something that makes their solu-tion better.

Figure 1 shows the change in leaderand increase in predictive accuracy in theKaggle-NASA mapping dark matter com -petition. Within a week of the competition

launch, the benchmark based on morethan 10 years of physics research wasbeaten by a glaciologist. The sameasymptotic pattern occurs in all Kagglecompetitions and often with the samesurprising winners.

Participants are located all over theworld and often work in elds that are, onthe surface, completely unrelated to thesource of the data problem, such as thehedge fund trader who recently won aneducation technology challenge or the


Figure 1: Predictive accuracy of the Kaggle-NASAmapping dark matter competition over time.

Figure 2: The universe of Kaggle participants offersa diverse skill set.

w or o b d r

r d od o d o r r ro dd r d o

o d or , o d

b r o

rod d od ro j b r rr o r ,

b b r o d o

o b .


16/47



neuroscience student solving an air pol-lution problem (see Figure 2). Kagglesfocus on data competitions with real-timeleaderboard feedback brings out fresh tal-ent and drives objectively better results.

prEDICTINg ThE FuTurE OFprEDICTINg ThE FuTurE

Using competitions to crowdsourceanalytic talent does not mean that in-house analytics groups become obso-lete, but it does change how they willapproach their role. In the future, themost valuable skill may not be solvingthe analytics problem itself, but being

able to clearly define and structure theproblem into a competition format andevaluate the results to determine whichmodel can be most successfully appliedto your particular business. By introduc -ing open, competitive research into theanalysts toolset, an opportunity arisesto leverage a large analytics talent poolin a cost-effective, scalable way.

Kaggles own expansion shows howcompetitive research can be applied tomany other points along the analyticsvalue chain. In the pas t year, Kaggle has

introduced specially formatted, invita-tion-only competitions in which a selectgroup of past competition winners areinvited under non-disclosure agreementto work on sensitive datasets. Anotherrecently introduced type of competition,marketed as Kaggle Prospect, extendsthe crowdsourcing model to the prob-lem identification phase that happensbefore the competition can be defined.Hosts share a sample of their raw datawith Kaggles participants, who can usethe data to suggest predictive modelsthat could be built, uncovering newor less obvious questions that further

analysis of the data can explore. At theother end of the analytics value chain,Kaggle makes it simple to operational-ize the best predictive models from thecompetitions and integrate them intoexisting systems by hosting and imple -menting the model and making it acces-sible to the client through an applicationprogramming interface.

Competitive research has alsocreated a reputation engine for theanalytics industry. Recently, Kaggle in -troduced recruiting competitions, which

allow companies to filter their data sci-ence applicants on demonstrable skillsrather than resumes. Instead of leafingthrough a stack of resumes and thenusing 10-minute technical brain teasersto determine if an applicant is skilled atsolving large, open-ended data prob-lems, a recruiting competition allowsorganizations to try before they buy.The first competition of this type waslaunched for Facebook and is one of themost popular contests that Kaggle hashosted. Competitive research, whetherbased on past competition results ora recruiting competition for a specific

company, in an objective and unbiasedway to filter talent. Screening applica -tions through a data science competi-tion decreases both the false positiveand false negative rates no more badinterviews and no more having greatcandidates waiting forever for the phoneto ring because they come from a non-traditional background.

Another impl icat ion of compe titi veresearch is in how analysts are com-pensated. A hedge fund trader who isgreat at predicting risk makes millions,so why is an equally smart analyst in adifferent industry, who is also great atpredicting risk, not compensated nearlyas well? In most industries, predictivemodeling is still tr eated as a cost center,

even if the results of the model have adirect impact on the companys profit -ability. The demand for people with theskills to crunch large amounts of datafar exceeds the supply, but salaries aremuch more sticky. The creation of com -petitive marketplaces for analytics isseeing the size of the prize pools startto converge toward the business valueof the models as the hosts bid up prizemoney to attract the best talent to theirproblem.

CONCLuSION

Selecting the best predictive model

through a research competition is, asSloan School Professor Andy McAfeeso eloquently describes it, not pickinga horse, but hosting a horse race. Dataresearch competitions are a resource-ef cient way for organizations to solvecomplex data problems, and they create ameritocratic market for talent that chang-es the way analysts work. As the crowd -sourced model for competitive analyticscatches on, companies have less needto build large, in-house analytics teams,but the people they do hire need to un-derstand how to structure their questionsfor competitive research. May the bestmodel win.

Margit Zwemer ([email protected]) is a datascientist and community manager at Kaggle.

R q o-ob o infORms m b r B pFor more information, visit: http://www.informs.org/Membership


17/47


co bor or b o o r o ro r od , d r do d or o , r - r r or ro .

Forecasting is the foundationof virtually all business plan-ning. Scheduling productionrequires a demand forecast.

Deciding between short- and long-terminvestments requires a cash ow forecast.

You need a customer usage forecast inorder to buy enough servers to supporta growing cloud-based software applica-tion. The list of forecasts that a companymust prepare is endless and, moreover,the forecasts are intertwined.

One of the biggest problems whenforecasts are inextricably linked is the lackof communication between stakeholders.No one knows which assumptions havebeen incorporated, whether they are upto date or which data is relevant. The re -

sult is incomplete information and poten-tially obsolete forecasts. This uncertaintyleads to confusion and mistrust in theforecast. Stakeholders start looking foralternative forecasts they can con dentlybase key decisions on; sometimes this

means building their own models, spread-sheets or local data systems. Other timesit means relying on their experience andgut-feel alone. While they may feel morecon dent, they still have no assurancethat data and assumptions are correct

and up to date.Because forecasting is so pervasive

and so important, it should be treated asthe plumbing application that it truly is.Imagine if all forecasting activities wereintegrated into a single, living enterprise

forecast. Data is automatically analyzedas soon as it becomes available. Peoplewith relevant insight about the businessand knowledge about planning decisionsinput this information as soon as it chang-es. The impacts of changes to data and

assumptions automatically ow throughthe system in real time, updating all rel-evant forecasts. You can see the impactson all areas of the organization as chang -es happen. You can test and understandthe implications of what-if scenarios and

co bor or :ro o o r

By BRian lewis

F

B u s i n e s s p l a n n i n g



18/47

17 | a n a ly t i c s - m a g a z i n e . o r g

c O l l a B O R a t i v e f O R e c a s t i n g

A leading marketing companys scoring models used to take4.5 hours to process. Now, with high-performance analyticsfrom SAS, theyre scored in 60 seconds.

Get the relevant insights you need to make decisions in anever-shrinking window of opportunity and capitalize on

the complexity of big data to differentiate and innovate.

269 What would you do with an extra 269 minutes?High-Performance ComputingGrid ComputingIn-Database AnalyticsIn-Memory Analytics

Big Data

high-performanceanalytics A realgame changer.

sas.com/269 for a free white paper

EachSAScustomersexperienceisunique. Actualresultsvarydependingonthecustomersindividualconditions.SASdoesnot guaranteeresults,andnothinghereinshouldbeconstruedasconstitutinganadditionalwarranty.SASandall otherSASInstituteInc.productorservicenamesare registeofSAS InstituteInc.in theUSAand othercountries.2012 SASInstituteInc.Allrightsreserved.S91566US.0412

alternative decisions. You can respondand modify your plans in real time.

Collaborative forecasting applica-tions are turning this vision into reality.These technologies are based on a fewfundamental concepts: balancing ana -lytics and insight, real-time updates ofdata and assumptions, and opennessand transparency.

BALANCINg ANALyTICS AND INSIghT

Every forecasting process falls on aspectrum that is bounded by two extremeapproaches pure analytics on one sideand pure insight on the other. Collab -

orative forecasting combines both ap-proaches, incorporating the best aspectsof each.

Analytics-based forecasting relieson forecasting engines in which sta -tistical and other mathematical modelsare applied to data to automatically ndtrends, patterns, seasonalities, outliers,shifts, correlations, etc. Essentially youpush data in and forecasts come out.These engines employ analytics that

nd and extract information from datathat humans could not easily nd them -selves (if at all). Computing power is dirtcheap these days, and automating theanalysis of massive amounts of data ina short period of time is extremely costeffective.

However, analytics-based forecastsare only as smart as their underlyingmodels and algorithms and are highlydependent on the input data. Forecast -ing engines cannot nd information

such as the effects of a promotion unless that information is somehowrepresented in the data and the mod-el knows to look for it. The data itselfmight be nonexistent (e.g., no historicalsales for a new product) or irrelevant(e.g., historical sales do not match acustomers new buying pattern). In anyof these cases, analytics-based fore-casting is not an option.

In contrast, insight-based forecastingrelies exclusively on human judgementand gut-feel. Key stakeholders applytheir experience, expertise, knowledgeof business operations, expectations ofcustomer actions, sense of market con-ditions, etc. to project future outcomes.This approach has bene ts, such asthe incorporation of timely rst-handknowledge, but it has aws as well.People are not particularly adept atanalyzing data for trends, patterns, etc.and therefore miss out on this contribu-tion to the forecasts. The approach isnot scalable or easily replicable, whichis a problem if your company needs toforecast many items. People are alsosubject to many sources of conscious


19/47



and subconscious biases, such as theoptimism of a sales team, which can in-appropriately skew forecasts.

Collaborative forecasting softwarecan balance analytics and insight byapplying analytics to find good base-line forecasts and then adjusting andoverriding them based on insight. Foreach item being forecasted, collabora-tive forecasting applications must allowyou to select the right mix and applica -tion of analytics and insight. As anyonewho has worked with large enterprisesoftware applications can attest, flex -ibility is a rarity. Fortunately, new col -

laborative forecasting applications aredesigned more as forecasting platformsthan forecasting tools and offer usersthe high degree of flexibility that theyrequire.

rEAL-TIME upDATES OF DATA ANDASSuMpTIONS

As a concept, collaborative forecast -ing is a living process in which fore -casts are always up to date. As soon assomeone changes an assumption or datais updated, the forecasts should incorpo-rate the changes across all departmentsin real-time. If the forecasts do not re ectthe latest information, then any decisionsyou base on those forecasts are immedi-ately obsolete.

Collaborative forecasting therefore re-quires a central forecasting applicationthat pulls in data from disparate systemsspread throughout the organization. Theapplication must provide mechanismsfor key stakeholders and knowledge-able people to record their assumptionsand link them directly to the forecasts.The application cannot simply documentand store insights in some spreadsheetor database. Instead, assumptions mustbe modeled quantitatively and linked to aforecast. As the assumptions change, theforecasts change in response.

For example, suppose your marketing

department plans to run a new promotionthat they believe will lift demand by 10percent next month. Rather than writing adocument or sending an e-mail with thesedetails, the marketing department shouldbe able to input key assumptions in thecollaborative forecasting application,such as the lift percentage, the launchdate and duration, the type of adjustment(i.e., multiply the lift by the baseline de -mand), and which forecasts the promo-tion should adjust. If marketing decidesto delay the promotion by one month, thedemand forecast will automatically re-

ect this shift as soon as they update thelaunch date.

Even simple decisions such as de -laying a promotion can have widespread

operational and financial implications.The cost to implement the promotionshifts by one month, which affects finan-cial forecasts. Did you need additionalheadcount to support the promotion? Ifso, this affects headcount forecast. Theshifted promotional demand impactsyour revenue forecast and potentiallyprocurement, inventory and productionforecasts. Remember, all forecasts areintertwined.

With real-time updates of data and as-sumptions, collaborative forecasting al-lows you to see the impacts of changesas they happen and be more responsive

in planning.

OpENNESS AND TrANSpArENCy

People trust processes that aretransparent and inclusive. Rather thantracking down and reconciling informa-tion from Marketing, Sales, Operations,Finance or HR, let them take directownership of the information they knowbest and link to the information that th eyneed. Collaborative forecasting allowseach stakeholder to add new informa-tion, propose changes, test scenariosand share insights in a single, sharedsystem. Their rationale and contribu -tions are tracked, time stamped anddocumented for complete opennessand transparency.

s b r b oAnalytics


w r -d o d

d o ,o bor

or o oo o

d b or r o

.


20/47

19 | a n a ly t i c s - m a g a z i n e . o r g

IMpLEMENTINg COLLABOrATIvEFOrECASTINg

Three potential hurdles exist to im -plementing a collaborative forecastingprocess: technology, awareness and read -iness. Collaborative forecasting softwarenow exists in many forms. Some of thenewest and most promising are compre-hensive and support all of the fundamentalconcepts of collaborative forecasting. Oth -ers are more limited, for example focusingmore on the analytics and the forecastingengine approach. With new technologycomes new awareness of what is possibleand the interest in collaborative forecast-

ing is increasing rapidly. However, evenwith technology and awareness, somecompanies may feel they are not quiteready to implement a collaborative fore-casting process. You know your processisnt great and your forecast accuracy isonly so-so, but process changes can bechallenging at your company. That maybe true, but the nancial and competitiveadvantages of collaborative forecastingare substantial enough that you cannotafford to wait any longer.

Brian Lewis, Ph.D. ([email protected]), isvice president of Professional Services for VanguardSoftware, an enterprise forecasting software companywith more than 2,000 customer in 60 countries,including 33 of the Fortune 100. He is a seniorINFORMS member.


n n n n n n n n n n n n n n n n








Decision Sciences Instituteadvancing the science and practice of decision making

The Decision Sciences Institute is a nonprotprofessional organization of researchers, managers,educators, and students interested in decision-making techniques and processes in private andpublic organizations. The Institute is an internationalorganization with over 3,500 members in 32 coun-tries. The annual meetings and regional conferencesattract over 3,500 participants a year.

The Decision Sciences InstituteIs Committed to . . .

Research. A focus on the integration of researchin the art and science of managerial decision makingacross traditional functional academic disciplines; aninternational forum for presentation and discussionof research.

Teaching. A forum for presentation and discussion

of innovative teaching; recognition of teachingexcellence and curriculum innovation.

Practice. An exchange of ideas between leadingprofessional practitioners and educators.

Benets Members Receive:

Annual meeting. Members receive discounted fees toattend, and the meeting draws over 1,500 professionalstogether to share current thoughts on theoretical andapplied issues.Decision Sciences is a highly respected journal amongscholars and is subscribed to by over 1,000 libraries.It seeks and publishes high quality, theoretical andempirical articles.Decision Sciences Journal of Innovative Education isa high quality, peer-reviewed scholarly journal whosemission is to publish signicant research.Decision Line , the Institutes news publication, ispublished ve times annually and includes featurecolumns, as well as information on members, regions,annual meetings, and placement activities.

Job Placement Services are offered throughout theyear, with position and applicant listings available viathe Internet. Also at the conference, special facilitiesaid in position search activities.

www.DecisionSciences.org

Decision Sciences Institute

43rd ANNUAL MEETINGNOVEMBER 17-20, 2012

Join us.Log on to our website and complete

the membership application to begin

enjoying your benets.

Regular membership: $160/year

Student membership: $25/year

Documenting the rationale for as -sumptions is a critical form of commu-nication between stakeholders. Whilethere still might disagreement aboutthe numbers themselves, stakeholderscan have more productive discussionsif they know the rationale behind them.Perhaps this leads to changes, perhapsnot. But access to information and opendiscourse builds trust in the process.Without it you run the risk of introduc-ing uncertainty, confusion and mistrust,and this is something you clearly wantto avoid.

Another reason for documenting ra -

tionale is for performance management.Everyone knows why they are updatingtheir assumptions today, but they maynot remember the details in a few days,let alone a few weeks or months fromnow. By time-stamping your changesand documentation, you have an audittrail of every change that was made.When you look back at older forecasts,you can see the old data and assump-tions, who was responsible, their ratio-nale for making changes and so on. Byknowing what was done and whetheror not it was successful, you can de-sign forecast accuracy and process im-provements that incorporate the goodand eliminate the bad.


21/47


s r o or o r r r r d d d o .

Time series data is frequentlycollected and analyzed, andthen forecasts are made, oftenwith clever headlines or leads.

These vary from the important to the mun-dane. Recently, we have seen time series

data and forecasts for unemployment gures,gas prices, mortgage loan rates, stock indi-ces, approval values for President Obamavs. Mitt Romney, high school science andmath scores, television ratings for DancingWith the Stars and many other examples.

Television and print commentatorsfrequently show the numbers or graphs,draw some general conclusion or predic-tion, but frequently they do not delve intosome underlying issues, such as samplesize, likely voters vs. registered voters,

correlations with other factors, forecastsare long-range and others. Many statisti -cal and forecasting software products willautomatically make forecasts for time se-ries data, but did the forecaster know thespeci cs of the model chosen, examine

previous forecasts made on earlier dataor try alternative models? We presumethe answer is absolutely, but sometimeswhen we read or hear about the forecasts,we wonder.

For the purposes of this article, I tried

making some forecasts on data that is farless weighty than many of the examplesgiven above. I collected [1] monthly dataof the total domestic motion picture box-of ce grosses, from January 2000 throughDecember 2011, or 144 values, and put

these into an Excel spreadsheet. Eachmonth had just three columns time peri-od, date (1/1/2000, 2/1/2000, etc., whichExcel then displayed as Jan-00, Feb-00,etc.) and the total gross (the numbers arein millions of dollars and have not been

adjusted for in ation). The plan was toimport the spreadsheet into several sta-tistical or forecasting software productsto make forecasts of the total box-of cevalues for January through April 2012,and then compare those forecasts to the

for rd r d?

By Jack yuR kiewicz

T

f O R e c a s t i n g s O f t w a R e s u R v e y

2 0 | a n a ly t i c s - m a g a z i n e . o r g a n a ly t i c s | J U ly / a U g U s t 2 012


22/47


23/47

w w w . i n f o r m s . o r g2 2 | a n a ly t i c s - m a g a z i n e . o r g a n a ly t i c s | J U ly / a U g U s t 2 012

Bayesian Information Criterion (SBIC),mean square error (MSE), etc.) andthen finds the optimal parameters for themodel, calculates forecasts and confi-dence intervals, gives variou s summarystatistics and makes graphs. The usercan override the recommended proce-dure, specify some other forecastingtechnique, and the software then findsthe optimal parameters for that model,gets the forecasts, etc. Most dedicatedforecasting products can operate in the

automatic mode; fewer general statisti-cal products (e.g., IBM SPSS Statisticsand Statgraphics) can be put into thiscategory.

The next automation level is semi-automatic. Here, the user specifies theparticular forecasting methodology thesoftware should use, for example, aBox-Jenkins model, and the softwareproceeds to find the optimal parametersof that model, the forecasts, summarystatistics, graphs, etc. Most generalstatistical products operate in the semi-automatic mode.

Finally, the third level of automation,

dubbed manual, requires the user tospecify both the model and the param-eters of that model. Thus, for manualsoftware, if the user specifies Wintersmultiplicative model, he or she mustalso enter the three smoothing con-stants. The software will then give theforecasts, plots, summary statistics,etc. After examining the summary sta -tistics, forecasts, etc., the user manu -ally enters new model parameters andrepeats the process. Finding the op -timal model parameters can thus be -come a tedious trial-and-error process.The standard advice still holds: If youfrequently forecasting time series data,you should consider using an automaticor semi-automatic product.


wOrkINg wITh ThE SOFTwArEThis article is not meant to formally re-

view any product, but I used the month-ly box-of ce data in the latest versionsof SPSS, Forecast Pro, Statgraphics,NCSS, Minitab and Systat, all of which Ihave used for some time. These productsrepresent a cross-section of the threeautomation categories described. Allthe products read and imported my Ex -cel spreadsheet, usually but not always,without additional tweaking (ForecastPro needs, in separate cells, informationabout time span, seasonality, etc.).

All the programs I tried were easy to

use. Menu systems are clear (I adheredto the time honored adage of not click-ing on Help unless and until it was a lastresort) and the output was easy to readand interpret. If the software had an au -tomatic mode, I always used that rst.Looking at the monthly box-of ce data inFigure 1, repetitive peaks and valleysappeared over the years. The data showshigh box-of ce values from May throughJuly, and secondary peaks in Novemberand December. Low box-of ce monthsare typically in February and September.These rules, off course, do not alwaysapply (e.g., December 2009), but thereappears to be monthly seasonality andan upward trend to the numbers. Thus,a Box-Jenkins or a Winters exponentialFigure 2: Forecast Pros recommendation.

Figure 3: Overriding Forecast Pros recommendation and specify-ing a Box-Jenkins model instead.

smoothing model could be two method-ologies to apply to this data. Forecast Prorecommended Winters method with ad -ditive seasonality (see Figure 2). WhenI overruled that advice and asked for aBox-Jenkins model, it recommended an

ARIMA(1,0,1)x(0,1 ,2) model (see Figure3). IBM SPSS Statisitics recommendedan ARIMA(2,0,12) model. The third auto -matic product, Statgraphics, after I told itto analyze all the models available and

nd the one with the smallest AIC, hadits StatAdvisor recommend an ARIMA(0,1,1)x(2,1,2)12 model in which a mul -tiplicative seasonal adjustment was ap-plied (see Figure 4).


24/47


FOrECAST rESuLTS AND A CAvEAT As mentioned in previous forecasting

surveys, a particular data set could resultin automatic or semi-automatic softwarerecommending different models and/ordifferent parameters for those models.Figure 5 shows the box-of ce forecastsobtained from some of these products.Because SPSS and Statgraphics bothrecommended Box-Jenkins models, I toldForecast Pro to utilize its recommendedprocedure, an exponential smoothingmodel, and then nd the appropriate Box-Jenkins model and make forecasts forboth. As NCSS does not make a model

recommendation, I speci ed it should usethe Box-Jenkins procedure and nd theappropriate parameters. The last columnshows the actual box-of ce totals for the

rst four months of 2012. For March 2012,all the products gave forecasts that weremuch lower than the actual box-of ce g -ure. Perhaps The Hunger Games, re -leased on March 23 and which grossed$233 million in just those nine days ofMarch, may have something to do withthose low projections.

On May 15, ABC News reported, Today, just over a third of U.S. adults are obese. By2030, 42 percent will be, says a forecast re-leased Monday. Thats not nearly as manyas experts had predicted before the once-rapid rises in obesity rates began leveling

off [2]. The article intimated that the Cen -ters for Disease Control and Prevention hadmade the forecast. Thus, we are getting aforecast of the fraction of Americans whowill be considered obese 18 years fromnow, and thats a lower fraction that wasforecast who knows when. In my box-of ceexample, all I wanted to do is to make box-of ce forecasts four months into the future,and our forecasting software substantiallyunderestimated the box-of ce grosses thatwould occur in month three. Perhaps whenwe read about such long-range (and evenshort range) forecasts, or contemplatemaking them ourselves, we might pause

and think The Hunger Games.

ThE SurvEy

For this years forecasting softwaresurvey , as in the past, we tried to identifyas many forecasting products as possible.We e-mailed the vendors and asked themto respond to our online questionnaire soreaders could see the features and ca-pabilities of the software. The purpose ofthe survey is to inform the reader of whatis available. The information comes fromthe vendors, and no attempt was made toverify the information they gave us. Inevi -tably, after the results are published, wehear, How could they have left out (my)product X! Thus, if we did make an omis -sion, we ask you to please accept our


Figure 4: Statgraphicss StatAdvisor making recommendation for the appropriate models to use. Only the top 2 choices are shown here.It also gives the parameters, the forecasts and some speci cs of the calculations for the optimal model (not shown).

Figure 5: Box-of ce grosses; forecasts from a sample of automatic and semi-automatic software and the actual grosses, in millions of dollars.


25/47


26/47


t o o r d o b d o ro ro , o d r d d b rror r o r d, o r .

A quick qu iz: What is a goodnine- or 10-letter descriptionof the emerging interest inbusiness analytics and big

data that ends in -al? A choice that may come to mind for

many is hysterical. This choice reflects

frenzied excitement about opportunitiesfor business analytics to solve problemsoften resulting from big data. Advocates

actually enthu siasts of analy tics havebecome energized by the growing inter -est in the fields of business intelligence

and data mining. But perhaps a less ob -vious choice of skeptical is an equallyvalid answer. Doubters and naysayersof business analytics believe the inter-est in these topics is overblown andmisguided.

whErE ArE ThE SkEpTICS COMINgFrOM?

Lets start with the view of the skepticsof analytics. Who are they? What is theirpro le? What is their objection to embrac -ing and deploying analytics?

a & b d : .

By gaRy cOkins

A

D R i v i n g c h a n g e



27/47


I will probably exaggerate, but here is mytake on the skeptics of analytics. They arelikely to be over the age of 50. When theytook a statistics course in college, many of

them probably just wanted to pass with aD and get the course behind them.

When the skeptics careers were shaped,they did not have PlayStations, Nintendosor Xboxes no video games at all. Theydid not have 150 channels on cable, satel -lite TV, video movies or DVDs. They had nosurround-sound, no smartphones, no per-sonal computers and no Internet.

During the skeptics early careers, theyobserved and participated in company re -

ghting. This was actually fun for them. Itkept them busy. The assertive skeptics werecontinuously promoted to higher job posi-tions because of their quick wit and intuition.These types of managers are not self-servingsycophants and political players (althoughsome are). They are hard workers.

The older skeptics grew up watchingtelevision series like Ozzie and Harrietand Leave It to Beaver. At their of ces

since most skeptics have been white-collar workers most of their careers their co-workers and supervisors werentmuch different from Ozzie Nelson andWard Cleaver, the fathers in those TV se -ries. Ozzie and Ward shaped the skep -tics values and attitudes of what the dailyworkday would be like. After a pleasant

breakfast at home with the family, oneheaded to the of ce to shuf e papers, at -tend meetings and return missed phonecalls (from handwritten secretary notes).

The career experiences of skeptics didnot involve punctuated change and vola-tility as occurs in todays sped-up world.However, it was not easy for them. Theirorganizations werent static and frozen intime. Occasional new products and ser -vices were developed, and new types ofcustomers pursued. Some companiesmerged, divested and acquired othercompanies. Periods of transition followedthose events before the skeptics work -

weeks settled down to a normal pace. But just like after a passing sto rm, they even-tually returned to shuf ing papers, writingmemos and attending meetings.

Skeptics were not free from solvingproblems or evaluating opportunities.Those are eternal tasks in anyones work -day. What was different about the solu -tions developed when skeptics careerswere shaped is the solutions just werentthat elegant they were like Swanson TVdinners, oven-baked in aluminum trays.

rELIANCE ON B uFFErS TO prOTECTAgAINST ErrOrS

Here is an example of problem-solv -ing in manufacturing that the skepticsexperienced. If a suppliers shipment of

component parts was going to be late,that was OK. The assembling manufac -turer simply rescheduled their customersorders, and its shipment would likely be

late to the distributor or retail store. If thisresulted in an out-of-stock shortage whencustomers wanted or needed the nishedproduct, then their customer just dealtwith it and made do. The reasoning wasthat sometimes it rains, and you donthave an umbrella. And Murphys Law ofthe unexpected periodically sneaks up.

When skeptics careers were shaped,there were no crystal balls to predict thefuture. But they did not need them. In

manufacturing a forecast was somehowproduced every month or so, and it wasused to establish plans for determiningwhat and how much inputs to buy (e.g.,raw materials, component parts) and whattypes of resources to hire or purchase.

When skeptics careers were shaped,they survived using buffers to protectthem from errors and missed deliveryschedules. Buffers were the magic elixirthat kept problems from becoming largeror more painful. Skeptics relied on threetypes of buffers related to time, ow andresources. They would start things ear -lier to buffer expected nish dates, buildand stock extra inventory to buffer mate -rial shortages, and add more people andequipment to buffer capacity.

a n a l y t i c s & B i g D a ta

w r rr d,

r d b r oro ro rror

d d d rd . B r r

x r rob ro b o

r r or or .

h pro o Analytics

Its fast and its easy! Visit:http://analytics.informs.org/button.html


28/47

2 7 | a n a l y t i c s - m a g a z i n e . o r g

Lets fast-forward to today. Skepticsrecognize that the world has changed.It is much more volatile. But they stillchallenge the need to embrace deep or

advanced analytics. When their careerswere shaped they observed peoplemeasuring something with a microm-eter, marking it with a piece of chalkand cutting it with a swinging ax. Theybehaved this way too. Why be precise?Few organizations recognized the pen -alties and extra buffer-related expensesand investments that mitigated againstimprecision and errors. Inef ciencies,long delivery lead times and temporary

shortages were tolerated by both sup-pliers and their customers.

whErE ArE ThE ENThuSIASTS OFANALyTICS COMINg FrOM?

Lets look next at the opposite view that of the analytical enthusiastswhose careers have been more re-cently shaped. Who are they? What istheir pro le? What is their enthusiasm

sometimes hysterically so for em-bracing and deploying analytics?

Again I will probably exaggerate tomake my points, but here is my takeon the enthusiasts. They are likely tobe under the age of 40. When theytook their university statistics courses,they had hand calculators and laptop

computers. (Believe it or not, in my rsttwo years in college we only had sliderules. I graduated in engineering andoperations research in 1971.)

During enthusiasts careers, re -ghting was not an occasional need it

was ongoing and never-ending. And itsintensity is not just because there aremore problems (although there are). Itis because there are more opportuni-ties requiring a sense of urgency.

These younger enthusiasts grew upwatching MTV, Friends and Seinfeld.Their sense of humor is more wry andcynical compared to the skeptics. Their

curiosity about how things work is prob-ably comparable to the skeptics whenthey were the same age. However, thedifference is the enthusiasts have fargreater ability to investigate and ana-lyze and with more computing powerand more functional software.

In the movie Moneyball, Brad Pittplays the role of the Oakland Athleticsbaseball team general manager BillyBeane. Just before Pitt res the teamshead baseball scout, he says (para-phrasing), OK. My turn. When you visitthe homes of an aspiring young baseballplayer you tell his parents that he hasa good chance of being a major leagueplayer, you dont know. You dont know.He repeats that to make his point.


Save $225 if you are part of a group of 3 or more.

Accommodations at Disneys Contemporary Resortare available for $180 per night, per room.

Additionally, specially-priced Theme Park ticketsare available for registrants.

Disney

Learn more

August 30 31, 2012Disneys Contemporary Resort & Convention Center

www.disneyurl.com/DAOS2012

Walt Disney Parks & Resorts is pleased to announce


29/47


This also applies to organizations, es -pecially where skeptics dominate. Skep -tics may think they know, but do theyreally know?

ENThuSIASTS uSE ANALyTICS TOrEpLACE BuFFErS

Lets revisit the example of manufac -turing and distribution companies. As pre -viously discussed, in the past the skepti cssolved operational problems with buffers.But the careers of todays enthusiastshave been shaped by buffers that mustbe paper-thin. Buffers that the skepticsenjoyed to protect them from the impact

of problems are too costly today. Thesecosts and penalties include surplus in-ventory, excessively long production anddelivery lead times, extra equipment ca -pacity, poor customer service levels andmore employees than are needed.

Analytics is reducing the size of andreplacing the protective crutch of buffers.Today, the enthusiasts for analytics areimaginative and visionary. Their thinkingdoesnt stop with enterprise resource plan -ning (ERP) systems, which can schedulepart production and purchasing based onforecast product demand volume, supplierdelivery rates and assembly lead times. En -thusiasts are far more imaginative than that

plus they have the analytics and comput-ing power to be creative.

What enthusiasts do is think forward lywith probabilistic what-if scenario anal-ysis. They do not view product distribu -tion in a supply chain as a linear tree,

branch and leaf structure that sequenc-es parts like elephants trunk-to-tails ina circus all the way from production tothe customer. Enthusiasts see opportu -nities in an integrated network of partsand products that exist or can be madeanywhere. And they then perform itera -tive trade-off analysis of the interrelatedvariables in real time.

Enthusiasts know there are four in -terrelated variables that have complex

interrelationships: customer service lev -els, lead times, demand volumes andunit costs. Since suppliers have mea -sured and calibrated all of the process-ing times and consumption rates fromrecent past period data, they then knowthe relationships among all four vari-ables. The analytics enthusiasts vision isone where they start with a baseline casescenario projected into the future. (Mostare already using advanced demandforecasting techniques.) They then per -form what-if analysis. They change oneof the variables probabilistically asthe independent variable, and calculatethe impact on the other three dependentvariables. And then they change anothervariable and so on.

This gives enthusiasts power to answermany questions, such as, What is the ad -ditional inventory carrying cost if we wantto improve service levels from 97 percent

to 99 percent for our strategic customers?These capabilities are no longer a dream orvision. They exist today.

ENThuSIASTS CAN wIN B uy-IN FrOMThE SkEpTICS

Admittedly my pro les of skeptics vs. en -thusiasts are exaggerated. They are not po -lar opposites but rather people residing alonga continuum. But this does not remove thechallenge of creating a culture for analytics.

My experience is that an effective wayto drive change, overcome resistance andgain buy-in is through example. Enthusi -asts can be role models. Lead by example.Demonstrate what can be done and it willbe done.

Gary Cokins ([email protected]) is an internationallyrecognized expert, speaker and author in advancedcost management and enterprise performance and riskmanagement systems. He is a principal in businessconsulting involved with analytics-based enterprise performance management solutions with SAS, a globalleader in business intelligence and analytics software.He began his career in industry with a Fortune 100

company in CFO and operations roles. He then worked15 years in consulting with Deloitte, KPMG and EDS. Histwo most recent books are Performance Management:Finding the Missing Pieces to Close the Intelligence Gapand Performance Management: Integrating StrategyExecution, Methodologies, Risk and Analytics. He blogsat http://blogs.sas.com/content/cokins. He is an INFORMSmember.


tod b d b b r

b r- .B r

jo d o ro ro o

rob r oo ood . t o d

d ror , x

o rod o dd r d , x r

q , ooro r r d

or o rd d.

Join the Analytics Sect ion of INFORMSFor more information, visit:http://www.informs.org/Co

Documents

Analytics Julyaugust 2012