7/28/2019 The Power of Metrics.doc
1/4
The Power of Metrics:Select Dashboard Visuals Using Six Sigma
Design of
Experiments
Column published in DM Review MagazineAugust 2004 IssueBy Kent
Bauer
My previous three columns focused on the visual aspects of the
dashboard including the framework, thecolor palette and the graphs.
As you may recall, the framework alternatives alone included a
board ofdirector viewer, briefing book, balanced scorecard,
portfolio chart and operational dashboard. Twenty-five
potential graph choices could be selected from basic,
difference, distribution and specialized graph styles.With all the
possible variations to choose from, how does one determine the
preferred visuals to meet theneeds of a particular business-user
constituency?The performance management needs of the senior
executive are certainly quite different than those for
aline-of-business (LOB) or operations manager. Tailored
environments and "management at a glance" are
now the operational mantras of the day. The Six Sigma toolkit
includes an optimization technique knownas design of experiments
(DOE) which can streamline the selection process of the preferred
visuals. Bypresenting business users with combinations of
alternatives, DOE can statistically separate the preferredvisuals
from the less desirable options.
Let's start with the definition and process for DOE, and then
examine a case study of the selection ofdashboard visuals. Although
more statistical definitions exist, let's define DOE as: a
structured, organizedapproach for effectively and efficiently
exploring the cause-and-effect relationship between numerousimpact
variables (the Xs) and the resulting performance variable (the
Y).In our case, we can think of the Xs as the input parameters of
the dashboard (the framework, colorpalette and graph type) while
the Y is the resulting dashboard's "look and feel." Although DOE
designs canbecome quite sophisticated with exotic names such as
Box-Wilson, Plackett-Burman and Taguchi, we will
focus on the more basic "full factorial" design. This will
provide a fundamental understanding of DOE andthe process involved
without the statistical complexities of the more elaborate design
schemes.The full factorial design will help us to identify the most
important sources of variation and quantify theimpacts of the
important Xs on the output Y. In the following section, we will
walk through the steps
required to successfully implement a DOE with a real case study
that builds on our existing knowledge ofdashboard visuals.One of
the first steps in setting up a full factorial design is to select
the number of factors and levels thatneed to be evaluated (see
Figure 1).
Figure 1: Factor/Level MatrixStep 1: Select three factors to be
tested - framework, color and graph type.
Step 2: Select two different test levels for each factor.
A = framework (briefing book, portfolio chart)
B = palette color (vivid, subdued)
C = graph type (stacked bar, radar chart)
http://www.dmreview.com/toc.cfm?issueid=20059http://www.dmreview.com/authors/author_sub.cfm?AuthorID=32172http://ad.doubleclick.net/jump/dmreview.com/;abr=!ie;pg=textad;sz=370x115;ord=12272022?http://www.dmreview.com/toc.cfm?issueid=20059http://www.dmreview.com/authors/author_sub.cfm?AuthorID=32172
7/28/2019 The Power of Metrics.doc
2/4
This combination of factors and levels is often called a
"treatment." The number of treatments requiredcan then be
calculated with a mathematical formula (n k) where "n" represents
the number of levels and"k" represents the number of factors.
Therefore, in our example, nk = 23 or 8 treatments. This allows
allvariations of the levels to occur at least once so both the main
impacts and interaction impacts can becalculated.Step 3: Develop
treatment matrix to incorporate all variations of the selected
factors and levels (seeFigure 2).
Figure 2: Treatment MatrixThe interaction effects (+) or (-) are
determined by multiplying the (+) or (-) of the respective
maineffects. For example, the AB interaction effect (-) for
treatment #3 is the result of the A main effect (+)multiplied by
the B main effect (-). Keep in mind those old rules from algebra
where: (+) x (+) = (+); (-)x (-) = (+); (+) x (-) = (-); and (-) x
(+) = (-).In some cases, the interaction impacts (e.g., briefing
book and subdued color palette) may have moreimpact than either of
the factors individually.Step 4: Develop ratio ratings scale (from
1-10), with 10 as the preferred dashboard alternative and 1 asthe
least attractive dashboard visual.Because we will be focusing on
the performance management needs of the senior executives, we will
needto present the various dashboard options and record their
reactions on a 1 to 10 scale.
Step 5: Randomize the treatment order and select eight senior
executives to review dashboard visuals.Step 6: Record results from
testing in the last column (Run x) of treatment matrix (see Figure
3).
Figure 3: Treatment ResultsIn our example, we are assuming that
the numbers in the last column for Run x would be symbolic of
the
results for the eight executives interviewed and reflect the
average values for these executives.Step 7: The same result numbers
are then used to populate the entire matrix while retaining the
existing(+) and (-) signs from the treatment matrix.Step 8:Add the
values in each column and put the appropriate result in the "sum"
cell of the treatmentmatrix (e.g., for Factor A = +19).
7/28/2019 The Power of Metrics.doc
4/4
Check out DMReview.com's resource portals for additional related
content, white papers, books and otherresources.
Kent Bauer is the managing director, Performance Management
Practice at GRT Corporation in Stamford,CT. He has more than 20
years of experience in managing and developing CRM, database
marketing, datamining and data warehousing solutions for the
financial, information services, healthcare and CPGindustries.
Bauer has an MBA in Statistics and an APC in Finance from the Stern
Graduate School ofBusiness, New York University. A published author
and industry speaker, his recent articles and
workshops have focused on KPI development, BI visioning and
predictive analytics. Please contact
[email protected].
http://www.dmreview.com/portals/default.cfmhttp://www.dmreview.com/portals/default.cfmmailto:[email protected]://www.dmreview.com/portals/default.cfmmailto:[email protected]
