20

T O P O F A R T I C L EC O N T E N T S < P R E V I O U S P A G E N E X T P A G E >

Bulleti

nof

the

Am

eric

anS

ocie

tyfo

rIn

form

atio

nS

cien

cean

dTe

chno

logy

–Ju

ne/J

uly

2007

Special Section

Data-Driven Design:Using Web Analytics to Validate Heuristicsby Andrea Wiggins

Andrea Wiggins received her MSI from the University of MichiganSchool of Information. She is a web analytics practitioner. She isalso associate instructor for the University of British Columbia’sAward of Excellence in Web Analytics certificate program. Herwebsite is www.andreawiggins.com. She can be reached by emailat akwiggins<at>gmail.com

W eb analytics, the practice of web trafficanalysis, provides intelligence for marketersand executives responsible for proving return

on investment (ROI). However, web analytics’ greatestpotential lies in improving the online user experience. Whenanalytics data is shared with the design team, a subtler andmore sophisticated user-experience design can emerge.

Web analytic data is mined from site activity logs andcan offer a wealth of information about user behavior.Information architects, when interviewed about the webanalytic measures of greatest value to their craft, centertheir comments on its value in providing context fordecisions and heuristic assumptions. There are a variety of popular heuristics in common use, and here we discussways that web analytics can be applied to using RobertRubinoff’s user experience audit [1].

His four basic factors of user experience – branding,functionality, usability and content – constitute a heuristicevaluation framework. This framework aims to provide “a quick-and-dirty methodology for quantifying the userexperience,” rather than a fixed algorithmic approach or acompletely subjective one. Rubinoff’s audit is a flexibleassessment tool, with which the site development teamselects audit statements for each factor and uses them toevaluate how well the site serves its users. A selection ofaudit statements provides the basis for the discussion ofthe role web analytics can play in evaluating the validityof the assessment.

Using HeuristicsProviding a context for heuristic assessments is a

useful application of web metrics in a site redesignproject. Analyzing pre-existing web traffic data can yieldinsights into user behavior and measure how well a site

design meets user needs [2] [3] [4]. By comparing userdata from web analytics to an information architect’sheuristic evaluation, a basic form of validation can emergefor otherwise subjective performance measures [5].

Web analytics is not a silver bullet against subjectivity,however; web traffic measurement will never be able toreveal the complete user picture, and this limitation is agood reason to use web analytics in combination withother tools such as surveys, customer databases and usertesting to evaluate the experience design. Web analyticsreveals what users do, but not why they do it, so measuresfor evaluation must focus on actions that can be mostdirectly mapped to user intent. This focus is a significantchallenge, as many heuristics are subjective and abstract,but the value of this effort extends beyond the designproject. Creating a framework for measurement is criticalto future evaluation of the success and value of strategicbut intangible investments like information architecture.

Validating Rubinoff’s User Experience AuditTo provide context for design, Rubinoff’s user

experience audit presents useful parameters (statements)that information architects can easily assess. His exampleuses four broad, equally weighted categories with fiveevaluative statements in each. The most functional aspectof this tool is its complete customizability. In fact, thislevel of flexibility requires that care be taken so the auditstatements reflect the site owners’ and users’ goals.

Because every domain operates in a different context,information architects and web analysts will achievegreatest success by consulting on the evaluative pointsthat will be most verifiable and valuable for each website.By selecting success measures cooperatively early in theproject’s definition or discovery phase, the design andevaluation will be in alignment from the start. Both theinformation architect and web analyst will then be betterpositioned to prove the value of their services and assurethat the project’s focus remains on business and user goals.

Creating an appropriate analysis of site traffic forsupporting the information architect’s needs requires thegeneration of validation measures for the site, congruent >

Information Architecture

21

T O P O F A R T I C L EC O N T E N T S < P R E V I O U S P A G E N E X T P A G E >

Information ArchitectureB

ulleti

nof

the

Am

eric

anS

ocie

tyfo

rIn

form

atio

nS

cien

cean

dTe

chno

logy

–Ju

ne/J

uly

2007

Special SectionW I G G I N S , c o n t i n u e d

to the heuristic considerations that the informationarchitect uses. Since the heuristics or user experience auditstatements will vary among information architects andprojects, a selection of representative evaluative points areexamined here, drawn directly from three categories inRubinoff’s user experience audit:

n Branding• The site provides visitors with an engaging and

memorable experience.• Graphics, collaterals and multimedia add value to

the experience.

n Usability• The site prevents errors and helps users recover

from them.• The site helps its visitors accomplish common

goals and tasks.

n Content• Link density provides clarity and easy navigation.• Content is structured in a way that facilitates the

achievement of user goals.• Content is appropriate to customer needs and

business goals.

Ideally, the most relevant metrics or key performanceindicators (KPI) should be comparable both before andafter a site is redesigned, although direct comparisons areoften impossible after fundamental changes to site structureare implemented. By nature, a site redesign generates newpoints of measurement, typically enhanced by improveddata collection strategies, and only a handful of the site’sprevious KPI might still be directly applicable.

Branding. A popular yet extremely subjective measure ofthe online brand experience requires judging whether thesite provides visitors with an engaging and memorableexperience. Because direct measurement of brand value iselusive in any medium, the most direct measure of thisephemeral value is the ratio of new to returning visitors.The desirable return visitor ratio for a site will vary. Forexample, tech support sites typically prefer a low proportionof return visitors, but for an e-commerce site, a continualinfusion of new visitors is needed to ensure viability, evenwhen returning visitors have a higher conversion value. Toeffectively measure the online brand experience, the teammust identify the ideal proportion of return visitors as aKPI specific to the business goals of the site. Thedevelopment team should track this statistic over time toevaluate their ongoing success in providing a positivebrand experience [6] [7].

The length of the average and median site visit, in bothtime and pages viewed, also provides a level ofverification of the information architect’s assessment ofthe brand experience.Using the statistics for theproportions of site visitsof each particular visitlength, the design teamcan set a specific goal forengagement length as aKPI for ongoing analysisof site performance.

The ability to validatethe assessment of aninteractive element’s brand value is directly dependentupon whether the interaction was created withmeasurement in mind. Engaging online experiences likemultimedia games, Flash and AJAX are all measurable –as is the ratio of visitors returning to interact with themand those visitors’ behaviors – but only if the designincorporates JavaScript tagging to report key interactions.In a redesign, KPI for interactive site elements should beconsidered in the development of the experience design.For example, the length of time users spend with aninteractive element, whether averaged or examined byaudience segment, provides a concrete idea of how longusers are engaged by the application. The average numberof return visits would be a simple statistic to assess howmemorable the experience is.

Another branding element that Rubinoff suggestsevaluating is the benefit that graphics and multimedia bringto the experience. Aside from ensuring that the graphicsdo not make the pages load too slowly, measuring theexperiential value of graphics can be more difficult unlessthey happen to be clickable graphics leading the user tofurther content. Tools such as Crazy Egg’s heatmap overlayof visitor click patterns can immediately reveal whetherclickable graphics drive more traffic than text links.

Usability. Rubinoff’s usability statements echo several ofJakob Nielsen’s usability heuristics, and because webanalytic technology is often sold as a usability tool, thereis a temptation to believe that it can replace provenusability testing, which is not true. Traffic analysis, nomatter how reliable and sophisticated, is not able toprovide the thoroughness or level of insight that labtesting can achieve. However, by combining usabilitytesting with web analytics, customer satisfaction >

Because directmeasurement of brandvalue is elusive in anymedium, the most directmeasure of this ephemeralvalue is the ratio of new to returning visitors.

22

T O P O F A R T I C L EC O N T E N T S < P R E V I O U S P A G E N E X T P A G E >

Information ArchitectureB

ulleti

nof

the

Am

eric

anS

ocie

tyfo

rIn

form

atio

nS

cien

cean

dTe

chno

logy

–Ju

ne/J

uly

2007

Special SectionW I G G I N S , c o n t i n u e d

information and a/b optimization, usability testingexpenses can be cut dramatically, and purportedly withoutloss in quality. A/b optimization is using pair-wisecomparisons to force users to state preferences, as in theoptometrist’s familiar “Which is better – A or B?”

Error prevention and recovery is a commonly selectedpoint for evaluation, and information architects can easily

determine whether a siteis offering appropriateerror handling. It isharder to quantify thevalue that error handlingcreates for the site’susers. The most directmeasures are simpleproportions: thepercentage of site visitsincluding a 404 (file notfound) error, thepercentage encounteringa 500 (server) error and

especially the percentage of visits ending with an error. A quick check can reveal whether errors merit a furtherinvestigation, and, if we dig deeper, the analysis becomesmore interesting with the addition of navigation analysis.For example, examining the pages most commonlyviewed one step before and after a user error can allow thedevelopment team to literally recreate it so that it can beunderstood and remedied.

Another favored usability heuristic asks whether thewebsite helps its visitors accomplish their goals andcommon tasks. Analyzing task completion success rateshelps to determine whether a site meets this usability goal.Task completion is now a relatively straightforward set ofweb analytic measures achieved by applying scenario orconversion analysis to specific, ordered tasks andexamining leakage points and completion rates. Leakagepoints, the places where users deviate from the designedprocess, should be of particular interest: when visitorsleave a process unexpectedly, where do they go, and dothey eventually return?

The straightforward percentage calculations thatcomprise conversion analysis are simple to determineonce the appropriate page traffic data has been collected.However, determining the significance of conversion ratesis another matter entirely. Conversion rate definitions andgoals are often KPI determined by committee, andunderstanding these measures will facilitate a deeper

understanding of the business and user goals that thedesign is intended to support. A primary difficulty oftenlies in choosing the tasks to assess; these measures arebest defined with the assistance and resultant buy-in of theparties who are ultimately responsible for drivingconversion. Shopping cart analysis is the most commonapplication of this type of evaluation; online formscompletion carries just as much weight for a leadgeneration site as checkout does for an e-commerce site.

Even the most common type of conversion analysis,shopping cart analysis, can be somewhatmisrepresentative of what it seeks to measure. Since asignificant proportion of shoppers engage in onlineresearch to support offline purchasing, a high shoppingcart abandonment rate may not be as negative as it wouldfirst appear. Another notable trend in shopping cartbehaviors is that customers often make use of theshopping cart as a calculator for their purchase totals andshipping rates, so frequent shopping cart item deletionscan also be misleading – perhaps the customer is simplyon a budget. Shopping cart analysis can prove very usefulfor determining which shopping tools and cross-marketing opportunities may add value.

Many other user behaviors can indicate the site’ssuccess in supporting user goals. Pogo-sticking, whereusers bounce back and forth between two levels of a site’sarchitecture, usually indicates that the organization of thecontent is problematic. Pogo-sticking behavior is anexcellent example of the benefits of collaboration. In thiscase the web analyst identifies the phenomenon and maybe able to infer some of the user needs behind it to sharewith the design team. The information architect, armedwith the knowledge of specific user behaviors and needs,can design a more successful, usable site [8] [9] [10].

Content. The evaluative points Rubinoff provides for thecontent category include simple queries about the site’snavigation, organization and labels. These foundationalcharacteristics of a site’s information architecture aresimply enormous territories for measurement [11]. Thereare many ways to attempt to measure these heuristics,and the applicability of any given measure is entirelydependent on context. Despite challenges in datacollection and analysis, navigation analysis findings aretypically too valuable to disregard, and they have a host of applications from scenario analysis to behavioralmodeling. One way to use navigation analysis findings incontext is to apply it to augmenting page traffic statistics.>

Leakage points, the placeswhere users deviate fromthe designed process,should be of particularinterest: when visitorsleave a processunexpectedly, where dothey go, and do theyeventually return?

23

T O P O F A R T I C L EC O N T E N T S < P R E V I O U S P A G E N E X T P A G E >

Information ArchitectureB

ulleti

nof

the

Am

eric

anS

ocie

tyfo

rIn

form

atio

nS

cien

cean

dTe

chno

logy

–Ju

ne/J

uly

2007

Special SectionW I G G I N S , c o n t i n u e d

Performing navigation analysis on the most popularcontent pages of the site shows the paths users traverse toarrive at and depart from those pages. Looking for trendsin visits to content versus navigation pages can also beindicative: if navigation pages are among the site’s mostpopular pages, there is probably good reason to spendsome time considering ways the site’s navigation mightbetter support user goals. Examining the proportions ofvisits using supplemental navigation, such as a sitemap orindex, can also reveal problems with primary navigationelements. In these cases, however, web analytic data ismore likely to point out the location of a navigationproblem than to identify the problem itself.

Determining whether content is appropriate to visitorneeds and to business goals is a complex problem. Tovalidate the information architect’s analysis of contentvalue for visitors, individual content pages or classes ofpages could be examined and compared on such measuresas the proportion of returning visitors, average pageviewing time length, external referrals to the page andvisits with characteristics indicative of bookmarking orword-of-mouth referrals.

Content group analysis is another common approach tomeasuring the value of website content. Site contentperformance is often measured by dividing the contentinto logical, mutually exclusive groupings and monitoringtraffic statistics and user behaviors within these contentgroups. The most useful content group analysis will sliceand dice the data across several sets of content groupings.These grouping may include audience-specific contenttracks, product-related content comparisons by numerouslevels of granularity (often presented in a drill-down reportthat allows a simultaneous view of every sub-level of acontent group) and site feature-specific content groupings.

To determine how well site content matches userexpectations, few tools can outperform search log analysis[6]. If analysis of site search query terms reveals asignificant disparity between the language that the sitevisitors use and that which the site employs, the chancesare good that the content does not fit user needs. The termsused to find the site in visits referred by search engines isanother source of information for comparing user needs tosite content, and these results should be considered inproportion to the volume of traffic that search enginesdeliver to a site.

A final heuristic from Jakob Nielsen merits mention:validating the match between the site and the real world.In Nielsen’s words, “the system should speak the users’

language, with words, phrases and concepts familiar to theuser…Follow real-word conventions, making informationappear in a natural and logical order” [12]. Closely relatedto the question of whether site content aligns with userneeds is the slightly different question of whether the sitevocabulary matches that of the site’s users. If the contentis right but the words are wrong, the site will havedifficulty attracting its audience.

A straightforward example shows the results of using amore common word in the navigation labeling versus amore site-centric word. In the original design of a regionallive professional theatre’s website, the site section forcontent related to actors, playwrights, staff and board waslabeled “Artists.” In a redesign of the site’s informationarchitecture, the original content and structure of the

FIGURE 1. When a page’s name was changed from “Artists” to “People,”visits to the page immediately increased by nearly 90% on average.A T-test of visit numbers over 13 days on either side of the name changeevent shows a significant difference, with a p-value of 0.0002.

content in the “Artists” site section was left intact, but thesection and its top-level page were renamed “People” inkeeping with the more common use of this contentcategory throughout the Internet. Figure 1 shows theimmediate impact of this change to visits to that page.

By comparing the user’s words to the website’s text andmining data from search engine visit referral terms andonsite search queries, web analytics can identify problemswith language and terminology. If the site has multipleaudiences with significantly different vocabularies, such asinvestors, doctors and patients, comparing search terms andsite text for the pages designed for these specific audiencesegments offers more targeted evaluation of whether thesite’s labels and content meet user expectations. The samesearch term analysis can also provide insight into what usersexpected to find on the site but did not. This helps >

24

T O P O F A R T I C L EC O N T E N T S < P R E V I O U S P A G E N E X T P A G E >

Information ArchitectureB

ulleti

nof

the

Am

eric

anS

ocie

tyfo

rIn

form

atio

nS

cien

cean

dTe

chno

logy

–Ju

ne/J

uly

2007

Special SectionW I G G I N S , c o n t i n u e d

identify business opportunities or gaps in search indexing.For this reason, null results deserve a more thoroughanalysis than is typically afforded, but at minimum, thefrequency of null search results should be monitored foranomalies, in keeping with general error handling analysis.

ConclusionWhen the web analyst and information architect are

evaluating the same qualities of a website, web analyticsuser data can serve as verification for informationarchitecture heuristics, even when the available methodsmay reflect only a few facets of a complex heuristic

concept or user experience audit statement. An informationarchitect’s heuristic evaluation of the user experience isoften subjective and gains value with a fact-basedunderstanding of the actual user experience. Access toweb analytic data during the earliest phases of a websiteredesign process informs the site’s architecture from thevery beginning, ideally allowing for a shorter engagementand resulting in a design that significantly improves thesite’s usability for its primary audiences’ needs. In thelarger picture, including measurement in site design helpsprove the return on investment of intangible investmentsin web presence.

For Web AnalyticsWeb Analytics Association. (Professional association for web

analytics practice) www.webanalyticsassociation.orgWeb Analytics Demystified. (“A site dedicated to making web

analytics professionals more successful.”) www.webanalyticsdemystified.com

Crazy Egg. (Free analytics tool for identifying location of clicks on a page) www.crazyegg.com

Google Analytics. (Free web analytics ASP using client-side data collection) www.google.com/analytics

Mentioned in the Article[1 Rubinoff, R. (2004, April 24). How to quantify the user

experience. Retrieved April 14, 2007, from http://www.sitepoint.com/print/quantify-user-experience

[2] Chatham, B. (2005). Integrated user profiles boost web analytics. Cambridge, MA: Forrester Research, Inc.

[3] Manning, H. (2004). Persona best practices: Developing your customer research plan. Cambridge, MA: Forrester Research, Inc.

[4] Sterne, J. (2002). Web metrics: Proven methods for measuring web site success (3rd ed.). New York: Wiley.

[5] Peterson, E. T. (2004). Web analytics demystified: A marketer’s guide to understanding how your web site affects your business (1st ed.): Celilo Group Media,CafePress. For further information see the Web AnalyticsDemystified Website (www.webanalyticsdemystified.com).

[6] Inan, H. (2002). Measuring the success of your website:A customer-centric approach to website management.Frenchs Forest NSW, Australia: Prentice Hall.

[7] Peterson, E. T. (2005). Webs site measurement hacks.Sebastopol, CA: O’Reilly.

[8 Beyer, H., & Holtzblatt, K. (1998). Contextual design:Defining customer-centered systems. San Francisco:Morgan Kaufmann Publishers.

[9 Holtzblatt, K., Wendell, J. B., & Wood, S. (2005). Rapid contextual design: A how-to guide to key techniques for user-centered design. San Francisco: Elsevier/Morgan Kaufmann.

[10] Nielsen, J. (2000). Designing Web usability. Indianapolis, IN.:New Riders.

[11] Rosenfeld, L., & Morville, P. (2002). Information architecturefor the World Wide Web (2nd ed.). Cambridge, MA: O’Reilly.

[12] Nielsen, J. (2005). Heuristics for user interface design.Retrieved April 13, 2007, from www.useit.com/papers/heuristic/heuristic_list.html

Resources